diff --git a/CHANGELOG.md b/CHANGELOG.md
index c1d73a6..0b52b42 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,66 @@ has a matching GitHub release whose body comes from its section below.
## [Unreleased]
+## [v0.4.1] - 2026-11-01
+
+Each card on a machine with several now has its own gateway URL, `/cards/CARD/v1`, and one model can be served
+on more than one card. A plain `/v1` request goes to the card that costs least to reach. `berth cordon gpu1` takes
+a card out of rotation while the others serve, and the gateway's `/gui/` page names each engine's card.
+`berth hold --gpu`, `berth vfio run --gpu` and `gpu = ...` on a `[[claim]]` hold only the cards they name. An
+engine can run under a systemd slice, the harbor shows the claims that match now, and an engine Berth stopped
+no longer shows as unresponsive.
+
+### Added
+
+- `GET /v1/berth/board` names the cards each engine's entry runs on (`cards`), and each card's base URL on the
+ gateway (`base`) and whether it is cordoned. On a node with several cards, the gateway's `/gui/` page names each
+ engine's card, so launching the copy of a model on gpu1 picks that card from a phone.
+- `berth cordon CARD` stops new grants on a card, holds aside, while the node's other cards serve; `berth uncordon
+ CARD` lifts it. What holds the card keeps running. Waiters say the card is cordoned, `berth status` marks it
+ `CORDONED`, and the gateway treats it as out of reach: no background engine starts there and a ComfyUI job for it
+ queues. A berthd restart clears the cordon.
+- Each card has its own base URL on the gateway, `/cards/CARD/v1`, named by the card's name or PCI address. A
+ request through it only reaches engines on that card, and `GET /cards/CARD/v1/models` lists only theirs. One
+ model can now be served on several cards, one engine entry per card, and `berth config check` still refuses two
+ entries that serve it on one card.
+- A plain `/v1` request for a model served on several cards goes to the card that costs least to reach: the engine
+ that is up, then one parked in RAM, then a free card, then a card whose holder drains, and last a card a VM or a
+ pinned lease keeps. Cards of the same rank go to the first entry in `berth.toml`.
+- `gpu = "gpu1"` on a `[[claim]]` holds only the cards it names, as `berth run --gpu` names them, on a node with
+ several cards. Engines on the other cards keep running, and the gateway starts background engines there. A
+ claim without `gpu` still holds every card. `berth config check` refuses a bad card name and a `gpu` on a
+ `track` claim. A name the node does not have is logged when berthd starts, and that claim holds every card.
+- `berth hold --gpu CARD` and `berth vfio run --gpu CARD` name the card on a node with several. `hold` takes a
+ comma list for one hold on several cards, and `vfio run` takes one card. Before, both always took the node's
+ first card. Without `--gpu` they still do.
+- `slice = "NAME.slice"` on an engine entry, or `berth run --slice NAME.slice` (which wins), starts the engine's
+ transient scope under that systemd slice on a Linux user manager: the slice unit's resource policy, CPU, memory
+ and swap limits, applies to the engine and everything it forks, not only to the wrapper that asked for the card.
+ `berth config check` refuses a slice name no unit can take. An entry with no slice starts its scope exactly as
+ before, on every platform.
+- The harbor's card tab has a CLAIMS panel: the claim rules matching now, as `berth claims` lists them. A hold
+ names its lease or says it is waiting, a track gives the card memory it names, and on a machine with several
+ cards each row names its card. Before, a tracked app such as OBS did not show in the harbor at all.
+
+### Changed
+
+- The harbor's About tab is rewritten for v0.4. It leads with the node's version and links to the docs,
+ changelog, issues and source, adds a section on machines with several cards, and drops the list of planned
+ features that had shipped or changed. Its sections are cards in two columns on a wide screen, with the same
+ section rail as the other tabs.
+- The harbor header no longer clips the ADVANCED switch or cuts its tagline off: below 1600 px the tagline
+ and the "& REQUESTS" half of the engines tab are hidden. On a phone an alert's message wraps instead of
+ ending in an ellipsis, with its time on its own line.
+
+### Fixed
+
+- An engine that Berth stopped no longer shows as unresponsive (LATCHED) in `berth status` and the harbor. A
+ drain, `berth stop`, a wake that failed or a stop signal from a supervisor such as llama-swap ends the engine on
+ purpose, and it now shows as offline, as an engine that exits 0 already did. An engine that exits non-zero on
+ its own still shows as unresponsive.
+- A harbor link to a section, such as `#sec-library` from the engines tab's rail, opened the card tab on a
+ reload. It now opens the tab that holds the section and scrolls to it.
+
## [v0.4.0] - 2026-10-29
One berthd now governs every card on a machine. `berth run --gpu gpu1` puts an engine on one card,
diff --git a/README.md b/README.md
index c3206c4..426dbfb 100644
--- a/README.md
+++ b/README.md
@@ -42,7 +42,7 @@ claim. When another engine holds it, Berth:
The numbered markers on the timeline are those four steps. A process that holds VRAM without a lease
shows up as a stowaway: Berth reports its PID and memory and leaves it running.
-> **Status: alpha, v0.4.0.** Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,
+> **Status: alpha, v0.4.1.** Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,
> Strata, ComfyUI and games, and testers run it on Windows. The [docs](docs/README.md) cover what works and has tests; the
> [roadmap](docs/roadmap.md) lists the phases and what is still missing.
@@ -108,7 +108,8 @@ line goes away.
stowaway report.
- **Your own rules.** `[[claim]]` in `berth.toml` covers anything else: a training run, a
render farm job, a cracking session, by name, command line or launcher.
-- **`berth hold`** frees the card for a call or a game you start yourself. Ctrl-C gives it back.
+- **`berth hold`** frees the card for a call or a game you start yourself. Ctrl-C gives it back. On a node with
+ several cards, `--gpu` names the ones it takes.
- **`berth run --pinned`** for an app or a training run that must never be stopped mid-work, on one card or
several. See [Training runs](docs/leasing.md#training-runs).
@@ -220,7 +221,13 @@ sits reserved while another is awaited. `CUDA_VISIBLE_DEVICES` lists each card's
on any of the cards stops the engine, and its exit frees every card. A lease on several cards cannot park or lend
yet: a drain stops it the usual way.
-The gateway judges each engine against the cards it runs on. "Any card that fits" is not built yet.
+The gateway judges each engine against the cards it runs on, and each card has its own base URL on the gateway,
+`/cards/gpu1/v1`, so a client can choose the card ([One URL per card](docs/gateway.md#one-url-per-card)). A plain
+`/v1` request for a model served on several cards goes to the cheapest card to reach: a parked copy, then a free
+card, then one whose holder drains.
+
+`berth hold --gpu gpu1`, `berth vfio run --gpu gpu1` and `gpu = "gpu1"` on a `[[claim]]` take only the cards they
+name, so the other cards keep serving.
## Quick start
diff --git a/docs/design.md b/docs/design.md
index 404c68d..7b9fe20 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -144,6 +144,6 @@ to undo.
| A2, shipped in v0.4 (experimental) | lends on one card through the protocol, with a stub engine | a sized borrower runs beside a lending holder; the return waits for the driver |
| B, shipped in v0.4 | one berthd for every card; leases with one share per card | a two-card engine leases both; the cards of a mixed box schedule apart |
| C, planned | parks and lends per share on multi-card leases | Maya on five cards lends from one card while the other four serve (#189) |
-| D, v0.5 | gateways joined over the tailnet | a request to one node runs on another and the leases stay per node |
+| D, v0.6 | gateways joined over the tailnet | a request to one node runs on another and the leases stay per node |
A stage that finds the model wrong changes this page first.
diff --git a/docs/gateway.md b/docs/gateway.md
index 7b5648f..0c52de2 100644
--- a/docs/gateway.md
+++ b/docs/gateway.md
@@ -278,7 +278,57 @@ curl -s http://127.0.0.1:9799/v1/berth/stop -H "Authorization: Bearer $(berth
free, held, draining, in a VM or failing its probe), then every engine in `berth.toml` with its state (up, offline,
waiting, starting, failed), whether it can be switched to (`launch`) or stopped (`stoppable`), and its models with a
`loaded` flag from the engine's own reader. `cards` lists every card the node governs, each with its name
-on the node (`card`, such as `gpu1`) and the holders on that card alone.
+on the node (`card`, such as `gpu1`), its base URL on this gateway (`base`, `/cards/gpu1/v1`), `cordoned` when
+`berth cordon` closed it, and the holders on that card alone. Each engine names the cards its entry runs on
+(`cards`), so a board for several cards can put each engine under its card.
+
+## One URL per card
+
+On a node with several cards, each card has its own base URL on the gateway: `/cards/CARD/v1`, where `CARD` is
+the card's name on the node (`gpu1`) or its PCI address. A request through it only reaches engines whose entry
+runs on that card, and `GET /cards/CARD/v1/models` lists only their models. Chat, completions, embeddings and
+`/v1/berth/wait` take the prefix.
+
+One model can be served on several cards, one engine entry per card, each on its own port:
+
+```toml
+[[engine]]
+name = "qwen-gpu0"
+type = "llamacpp"
+url = "http://127.0.0.1:8080"
+gpu = "gpu0"
+start = ["llama-server", "-m", "/models/qwen.gguf", "--port", "8080"]
+serves = ["qwen"]
+
+[[engine]]
+name = "qwen-gpu1"
+type = "llamacpp"
+url = "http://127.0.0.1:8081"
+gpu = "gpu1"
+start = ["llama-server", "-m", "/models/qwen.gguf", "--port", "8081"]
+serves = ["qwen"]
+```
+
+A client that works on gpu1 sets its base URL to `http://127.0.0.1:9799/cards/gpu1/v1` and keeps the model name.
+`berth config check` still refuses two entries that serve one model on the same card, or where either names no
+card.
+
+A request without the prefix picks the card itself. An engine that is up and lists the model answers first.
+Otherwise the gateway takes the card that costs least to reach, in this order:
+
+1. the card where the model's engine is parked in RAM;
+2. a free card: no lease, waiter, stowaway or active claim, and not in a VM;
+3. a card whose holder drains the usual way;
+4. a card a VM, a pinned lease (a game, a hold) or a cordon keeps, where the request waits.
+
+Two cards of the same rank go to the entry that comes first in `berth.toml`.
+
+- A card the node does not have answers `404` with the node's cards.
+- A cordoned card ([Cordon a card](leasing.md#cordon-a-card)) still takes requests through its URL, and they wait
+ in line.
+- A model no engine on that card serves answers `404` naming the card.
+- An engine berthd finds that has no entry in `berth.toml` belongs to no card here, so only the plain `/v1` URL
+ reaches it.
## Run an engine whenever the card is free
@@ -315,7 +365,7 @@ idle_after = "60s" # how long the card must stay free first (def
- A card back from a VM is a free card, so a background engine also picks up after a VFIO session.
- On a machine with more than one card, the card is the one the entry's `gpu` names (the first card when it names
none). A lease, a waiter or a stowaway on another card does not keep it out. A running claimed app does, because a
- claim holds every card.
+ claim holds every card unless its `gpu` names other cards.
- A background engine needs `start`, runs at `batch` only, and needs no `serves`.
An engine whose work needs a sidecar runs the same way as a Compose project. Set `compose` instead
@@ -385,6 +435,8 @@ Point the bookmark (or a home-page tile) at `http://127.0.0.1:9796/gui/comfyui`
- `/gui/` lists every engine in `berth.toml` with its live state. Any engine with a `start` command gets a Launch
button there, with or without a `gui` URL, so a phone on the tailnet can switch the card to Strata or a
llama-server too. An engine with no web UI shows **Running** once it answers.
+- On a node with several cards, each row in `/gui/` also names its card. With one engine per card for a model
+ ([One URL per card](#one-url-per-card)), the row you launch is the card you pick.
@@ -398,6 +450,22 @@ loopback, accepts loopback Host names plus the gateway's `--allow-host` names (f
front), and takes a start only as a POST from its own page, so another site cannot make a visitor's
browser drain the card.
+## Run an engine under a resource-policy slice
+
+`slice = "NAME.slice"` on an engine entry (or `berth run --slice NAME.slice`, which wins) places the engine's
+transient scope under that slice: the slice unit's CPU, memory and swap limits cover the engine and every child
+it forks, where the wrapper's own unit limits cover only the wrapper. The slice unit is your file: for example
+a `[Slice]` drop with `MemorySwapMax = 0` keeps a model-resident engine from paging at all. Berth refuses a slice
+name no unit can take before the engine starts; entries with no slice behave exactly as before, on every platform.
+
+```toml
+[[engine]]
+name = "llm"
+type = "llamacpp"
+url = "http://127.0.0.1:8080"
+slice = "llm.slice"
+```
+
## Drain a server at its request boundary
An engine with `/slots` but no unload URL (Strata, a bare llama-server) would otherwise get SIGTERM mid-request when
diff --git a/docs/harbor.md b/docs/harbor.md
index 5ce41b7..18d2a9a 100644
--- a/docs/harbor.md
+++ b/docs/harbor.md
@@ -146,8 +146,10 @@ question an operator actually asks.
in RAM), the queue rows, the container a run is in, and the exact terminal order
for the lease beside it as a copy button (`berth stop ENGINE`, `berth resume
ENGINE`, `berth reclaim LEASE`).
-- **The berths, the machine, stowaways and no ticket** answer "what else is on the
- card and the host, and who holds memory without a ticket?"
+- **The berths, the machine, claims, stowaways and no ticket** answer "what else is on the
+ card and the host, and who holds memory without a ticket?" The claims panel lists the claim
+ rules matching now, as `berth claims` does: a hold names its lease or says it is waiting, and a
+ track gives the card memory it names. Idle rules are only counted.
- **Several cards.** When berthd governs more than one card, each card has its own berth panel that lists
only the leases and stowaways on that card. Each lease line names the model its engine reports
loaded, and a lease on several cards adds its card list (`on gpu0..gpu4`). A board row and a stowaway name their card (`gpu1`), a lease
@@ -170,8 +172,10 @@ question an operator actually asks.
recent requests and context residency** answer the questions behind the request
log, and the storage panel (advanced) says what the harbor's own store keeps.
-**The ABOUT tab** is this page's own documentation: the modes, the shortcut keys,
-and what every state word and figure means.
+**The ABOUT tab** is this page's own documentation: the node's version, links to these docs and the
+changelog, several cards, the modes, the shortcut keys, and what every state word and figure means. A
+section rail jumps between its sections, as on the other tabs. A section link, from any tab's rail,
+opens the tab that holds the section when you share or reload it.
## Telemetry (built in, no other tool needed)
- **GPU:** memory, utilisation, temperature, power against its limit, fan, SM and memory
@@ -185,7 +189,9 @@ and what every state word and figure means.
"no window / no slot data", never an error. An engine that was up and stops
answering is shown as **unresponsive** (latched), distinct from one that never ran. A
`berth run` engine that exits 0 shows as offline, since a batch job that finishes stops
- answering on purpose. One that exits with any other code stays unresponsive.
+ answering on purpose. So does one that Berth stopped: a drain, `berth stop`, a failed wake,
+ or a stop signal from its supervisor. One that exits with any other code on its own stays
+ unresponsive.
- **Requests:** a 24-hour request log from llama-swap and Strata (a Strata entry needs
`type = "strata"`, and each request lands under the model its `/health` names); requests per minute, p50 and p95
latency, prefill time, time per output token, cache and re-ingest share, a token ledger
diff --git a/docs/interfaces.md b/docs/interfaces.md
index eda1955..c51cad0 100644
--- a/docs/interfaces.md
+++ b/docs/interfaces.md
@@ -1,7 +1,7 @@
# Interfaces
- **CLI:** one command tree with one output contract. `berth status`, `berth events -f`,
- `berth run`, `berth hold`, `berth stop`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
+ `berth run`, `berth hold`, `berth stop`, `berth cordon|uncordon`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
`berth wire llama-swap`, `berth engine check`, `berth config check`, `berth config show`,
`berth export`, `berth key`, `berth skill install|show`, `berth update`, `berth version`,
and for the node itself `berth daemon`, `berth harbor`, `berth service install|uninstall|status`,
diff --git a/docs/leasing.md b/docs/leasing.md
index 7a96734..11fa9d4 100644
--- a/docs/leasing.md
+++ b/docs/leasing.md
@@ -35,7 +35,8 @@
- **Several cards.** One berthd governs every card by default. A foreground request on one card never drains a
holder on another. `berth run --gpu gpu1` puts a lease on one card, and `--gpu gpu0,gpu1` holds one lease on
both (see [Training runs](#training-runs)). A running game holds every card, because Berth cannot tell which
- card it will use. A lease on several cards cannot park or lend yet.
+ card it will use. A `[[claim]]` with `gpu = "gpu1"` holds only the cards it names. A lease on several cards
+ cannot park or lend yet.
- **Fail-open, never silent.** If the daemon is down, `berth run` still starts the engine
but records it as an *unprotected run* that status shows (`--fail-closed` refuses).
- **Survives a daemon restart.** A running engine stays up when berthd restarts or is
@@ -106,6 +107,7 @@
cmdline = 'train\.py' # regex; on Windows it sees the exe's full path
action = "hold" # or "track"
linger = "30s"
+ gpu = "gpu1" # hold only these cards (names or PCI addresses, comma list); default every card
[claims]
builtin = false # turn the built-in pack off
@@ -117,12 +119,27 @@
`reaper SteamLaunch AppId=...`, so the REAPER DAW, also named `reaper`, holds nothing.
- **`berth hold [--for 2h] [--reason game]`** takes the card for you. It is a pinned lease at
the top priority, so nothing drains it, and it ends on Ctrl-C, after `--for`, or when the
- process dies.
+ process dies. On a node with several cards it takes the first one. `--gpu gpu1` holds another
+ card, and `--gpu gpu0,gpu1` holds both under one lease.
- **`berth run --pinned`** gives an app a lease that is never drained (it keeps the card until
it exits).
- A hold cannot free memory held by something outside Berth: that memory is still reported,
as contested.
+## Cordon a card
+
+On a node with several cards, `berth cordon gpu1` stops new grants on that card while the others keep serving, for
+maintenance or before a VM takes it. Name the card by its name or its PCI address.
+
+- What holds the card keeps running. A cordon drains nothing: `berth stop ENGINE` does that.
+- Waiters stay in line, and their reason says the card is cordoned. A hold (`berth hold`, a game claim) is still
+ granted.
+- The gateway treats the card as out of reach. A plain `/v1` request goes to another card that serves its model,
+ a background engine for the card does not start, and a ComfyUI job for it queues.
+- `berth status` marks the card `CORDONED`, and `berth events` records `card.cordoned` and `card.uncordoned`.
+- `berth uncordon gpu1` lets new work in, and the card's waiters are scheduled at once. A berthd restart also
+ clears the cordon.
+
## Training runs
A training run that loses its cards loses everything since its last checkpoint. Run it under one pinned lease:
diff --git a/docs/prerelease.md b/docs/prerelease.md
index 6e02568..8b5d47f 100644
--- a/docs/prerelease.md
+++ b/docs/prerelease.md
@@ -1,7 +1,7 @@
# Testing a pre-release
Work for the next release lands on the `next` branch before it reaches `main`. When it is ready for testers, it is
-tagged as a pre-release (`v0.4.0-rc.1`, then `rc.2` and so on) on the
+tagged as a pre-release (`v0.4.1-rc.1`, then `rc.2` and so on) on the
[releases page](https://github.com/T-Crypt/berth/releases), marked **Pre-release**.
Your normal install is not affected. The README's download link, `go install ...@latest` and `berth update` all stay
@@ -15,7 +15,7 @@ how it works. Back up `berth.toml` before you switch.
Pick the tag from the releases page, then on Linux:
```sh
-TAG=v0.4.0-rc.1
+TAG=v0.4.1-rc.1
cd /tmp
curl -fLO https://github.com/T-Crypt/berth/releases/download/$TAG/berth-linux-amd64
curl -fLO https://github.com/T-Crypt/berth/releases/download/$TAG/SHA256SUMS
@@ -31,7 +31,7 @@ it where your current `berth.exe` is, and restart Berth the way you started it.
Or build it with Go 1.27 or newer:
```sh
-go install github.com/T-Crypt/berth/cmd/berth@v0.4.0-rc.1 # a tagged pre-release
+go install github.com/T-Crypt/berth/cmd/berth@v0.4.1-rc.1 # a tagged pre-release
go install github.com/T-Crypt/berth/cmd/berth@next # the newest commit on next
```
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 1a2faf3..0a9958e 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -12,7 +12,7 @@
| P5.5 | Harbor router: one URL serves models from any node in the fleet | Planned |
| P6 | ComfyUI job-boundary handoff on Linux (**done**); ComfyUI started on demand through the gateway (**done**); Windows node (**experimental**) | Partly done |
| P7 | Offers: a waiting request can offer for an idle card on another machine; the holder decides | Planned |
-| P8 | Multi-GPU scheduling: a lease table per card, "any card that fits" | Started: one berthd for every card, an engine on one card or a lease on several (experimental) |
+| P8 | Multi-GPU scheduling: a lease table per card, "any card that fits" | Started: one berthd for every card, an engine on one card or a lease on several (experimental); per-card gateway URLs and cordon in v0.4.1 |
| P9 | Organisations: single sign-on, role-based access, audit log, Postgres, regional federation; a mobile app after | Planned |
| P10 | Opt-in auto-yield: a card's owner can pre-approve offers for their own card | Later |
@@ -93,7 +93,36 @@ v0.4.0 puts a whole machine under one berthd. Testers ran it first as pre-releas
The [CHANGELOG](https://github.com/T-Crypt/berth/blob/main/CHANGELOG.md) has every entry.
-## Next
+## Shipped: v0.4.1, a base URL per card (2026-11-01)
+
+v0.4.1 lets a client choose the card on a node with several. Testers ran it first as `v0.4.1-rc.1`.
+
+- **One base URL per card.** `/cards/CARD/v1` reaches only the engines on that card, and one model can be served
+ on several cards, one engine entry per card.
+- **Placement for a plain `/v1` request.** When several cards serve the model, the gateway picks one: the card
+ where it already runs, then one where it is parked, then a free card, then one whose holder drains, and last a
+ card a VM, a game or a cordon keeps.
+- **Cordon a card.** `berth cordon gpu1` stops new grants on the card, holds aside, until `berth uncordon`, so it
+ can be drained for maintenance or a VM while the others serve.
+- **A card picker on the board.** `GET /v1/berth/board` names each engine's cards and each card's base URL and
+ cordon, and the gateway's `/gui/` page names each engine's card.
+- **The card for holds, VFIO runs and claims.** `berth hold --gpu`, `berth vfio run --gpu` and `gpu = ...` on a
+ `[[claim]]` take only the cards they name on a node with several, so the other cards keep serving.
+- **Engines under a systemd slice.** `slice = "NAME.slice"` on an engine entry, or `berth run --slice`, puts the
+ engine's scope under that slice's CPU, memory and swap policy on Linux.
+- **The harbor.** A CLAIMS panel on the card tab, a rewritten About tab, and a header that no longer clips.
+- **Status after a stop.** An engine Berth stopped shows as offline, not unresponsive.
+
+## Next: v0.5, choosing the card from the harbor
+
+- **A SWITCH tab in the harbor** with a card picker, as the gateway's `/gui/` page has.
+
+Still to decide, with data from multi-card users:
+
+- Whether a cordon survives a berthd restart. Today it lives in berthd's memory, and a restart clears it.
+- Whether measured switch costs rank the cards for a plain `/v1` request. Today the gateway ranks them by reach.
+
+## Also queued
Each item has its building blocks merged or a spec written:
@@ -117,7 +146,7 @@ Not started, and in no release yet: `berth fit` and scenes, so engines that fit
| next | Parks and lends on a lease that holds several cards, one share per card ([Design](design.md)) |
| next | Broader AMD Linux hardware validation beyond the experimental RDNA3/RDNA4 probe; the Windows node out of experimental (a packaged installer, a locked-down control pipe) |
| next | Proxmox in every common layout. Berth detects whether it runs on a Proxmox host, in an LXC container or in a VM and says where it belongs (`berth env`). Engines in LXC containers lease the host's card, and a system service covers hosts without a user session |
-| later | Cards you name yourself, and "any card that fits" placement |
+| later | Cards you name yourself, and "any card that fits" for leases outside the gateway |
| v0.6 | The fleet: Berth on every machine, one harbor, one URL for every model |
| later | Offers: a waiting request can ask for an idle card on another machine, and its holder decides |
diff --git a/docs/vfio.md b/docs/vfio.md
index 79ce4e1..d9060ec 100644
--- a/docs/vfio.md
+++ b/docs/vfio.md
@@ -47,8 +47,9 @@ group and the driver bound to each function, the processes holding the card's de
berthd reports. While the card is bound to `vfio-pci`, `nvidia-smi` cannot see it, so pass the card's
address yourself (`berth vfio status 0000:01:00.0`).
-`berth vfio run [--reason NAME] [--before CMD] [--after CMD] -- CMD` holds the card and runs your
-script:
+`berth vfio run [--gpu CARD] [--reason NAME] [--before CMD] [--after CMD] -- CMD` holds the card and
+runs your script. On a node with several cards it takes the first one. `--gpu gpu1` (or the card's PCI
+address) passes another card through, and only that card's engines drain:
```sh
berth vfio run --reason galaxy \
diff --git a/internal/broker/broker.go b/internal/broker/broker.go
index 4732cb9..d18a8e2 100644
--- a/internal/broker/broker.go
+++ b/internal/broker/broker.go
@@ -479,6 +479,7 @@ type Broker struct {
saveFailed bool
verdicts map[string]verdict // loser engine key -> the juggle it lost
+ cordoned bool // berth cordon: grant only holds
// tracked is the claim watcher's last scan: a process (and every child)
// a track claim names. Its card memory is the claim's, never a stowaway.
tracked map[int]string
@@ -1337,6 +1338,9 @@ func (c cardBudget) avail() uint64 { return sat(c.ceiling(), c.held) }
// evaluate decides whether w fits now. If not, it returns why and which
// holders to drain; no victims means draining would not help, so w waits.
func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
+ if b.cordoned && w.prio != Hold {
+ return false, "the card is cordoned; berth uncordon lets new work in", nil
+ }
// A resumed lease is still in b.leases while it waits: its own
// residual is the copy it comes back to, never a holder it waits on.
hs := slices.DeleteFunc(b.holders(), func(h *lease) bool { return h == w })
@@ -1817,3 +1821,22 @@ func (b *Broker) abandon(l *lease, why string) error {
b.emit(proto.Event{Kind: proto.EvRefused, Lease: l.id, Engine: l.engine, Detail: why})
return fmt.Errorf("%w: %s", ErrRefused, why)
}
+
+// Cordon stops new grants on the card but holds (on), or lets them in again
+// (off). Holders keep the card; a cordon drains nothing. A change goes on
+// the event stream, and an uncordon schedules the queue at once.
+func (b *Broker) Cordon(on bool) {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ if b.cordoned == on {
+ return
+ }
+ b.cordoned = on
+ kind := proto.EvUncordoned
+ if on {
+ kind = proto.EvCordoned
+ }
+ b.log.Info("cordon", "card", b.cfg.Card, "on", on)
+ b.emit(proto.Event{Kind: kind})
+ b.schedule(b.now())
+}
diff --git a/internal/broker/cordon_test.go b/internal/broker/cordon_test.go
new file mode 100644
index 0000000..75e90f7
--- /dev/null
+++ b/internal/broker/cordon_test.go
@@ -0,0 +1,39 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "strings"
+ "testing"
+
+ "github.com/T-Crypt/berth/internal/procs"
+)
+
+// A cordoned card grants nothing new but a hold, keeps its holder, and
+// says why in the waiter's reason; uncordoning lets the waiter in.
+func TestCordonStopsNewGrantsUntilUncordoned(t *testing.T) {
+ b, _ := startTurnBroker(t, procs.Map{})
+ holder := must(t, acquire(b, Request{Engine: "llm", Priority: Batch}))
+ b.Cordon(true)
+ if st := b.Status(); len(st.GPUs) == 0 || !st.GPUs[0].Cordoned {
+ t.Fatalf("status does not show the cordon: %+v", st.GPUs)
+ }
+ holder.Release()
+
+ w := acquireW(b, Request{Engine: "image", Priority: Foreground})
+ pending(t, w.ch)
+ if r := firstReason(t, w); !strings.Contains(r, "cordoned") {
+ t.Errorf("reason = %q, want the cordon named", r)
+ }
+ hold := must(t, acquire(b, Request{Engine: "game", Priority: Hold}))
+ hold.Release()
+ pending(t, w.ch)
+
+ b.Cordon(false)
+ b.Tick(t.Context())
+ must(t, w.ch)
+ if st := b.Status(); st.GPUs[0].Cordoned {
+ t.Error("status still shows the cordon")
+ }
+}
diff --git a/internal/broker/status.go b/internal/broker/status.go
index e01e03c..93756f9 100644
--- a/internal/broker/status.go
+++ b/internal/broker/status.go
@@ -66,7 +66,7 @@ func (b *Broker) Status() proto.Status {
Util: d.Util, Temp: d.Temp, Power: d.Power, PowerLimit: d.PowerLimit, Fan: d.Fan,
ClockSM: d.ClockSM, ClockMem: d.ClockMem, PCIeGen: d.PCIeGen, PCIeWidth: d.PCIeWidth}
if d.Index == b.snap.Index {
- g.Scheduled, g.Baseline = true, b.snap.Baseline()
+ g.Scheduled, g.Baseline, g.Cordoned = true, b.snap.Baseline(), b.cordoned
g.UnobservedPIDs = append([]int(nil), b.snap.Unobserved...)
g.GTTBaseline = b.snap.GTTBaseline()
}
diff --git a/internal/claims/claims.go b/internal/claims/claims.go
index d79c574..348f008 100644
--- a/internal/claims/claims.go
+++ b/internal/claims/claims.go
@@ -25,6 +25,8 @@ import (
"regexp"
"strings"
"time"
+
+ "github.com/T-Crypt/berth/internal/gpu"
)
const (
@@ -51,7 +53,12 @@ type Rule struct {
Action string // hold | track
NeedsMiB uint64 // hold only: 0 = the whole card
Linger time.Duration // hold only: 0 = DefaultLinger
- Builtin bool
+ // GPU is the cards a hold takes, as berth run --gpu names them; "" is
+ // every card. Cards is GPU resolved to positions in the watcher's
+ // cards, nil for every card.
+ GPU string
+ Cards []int
+ Builtin bool
cmdline *regexp.Regexp
not *regexp.Regexp
@@ -110,6 +117,14 @@ func (r *Rule) Compile() error {
default:
return fmt.Errorf("claim %q: action %q (hold, track)", r.Name, r.Action)
}
+ if r.GPU != "" {
+ if r.Action != ActionHold {
+ return fmt.Errorf("claim %q: gpu names the cards a hold takes; a %s claim holds none", r.Name, r.Action)
+ }
+ if err := gpu.CheckCard(r.GPU); err != nil {
+ return fmt.Errorf("claim %q: %v", r.Name, err)
+ }
+ }
if len(r.Exe) == 0 && r.Cmdline == "" && len(r.Parent) == 0 {
return fmt.Errorf("claim %q: give exe, cmdline or parent to match on", r.Name)
}
diff --git a/internal/claims/claims_test.go b/internal/claims/claims_test.go
index 5718648..1bd99dc 100644
--- a/internal/claims/claims_test.go
+++ b/internal/claims/claims_test.go
@@ -7,6 +7,7 @@ import (
"context"
"io"
"log/slog"
+ "slices"
"strings"
"sync"
"testing"
@@ -77,6 +78,8 @@ func TestCompileRejects(t *testing.T) {
{Name: "x", Exe: []string{"["}},
{Name: "x", Exe: []string{"a"}, ParentCmdline: "SteamLaunch"},
{Name: "x", Parent: []string{"reaper"}, ParentCmdline: "("},
+ {Name: "x", Exe: []string{"a"}, GPU: "card 1"},
+ {Name: "x", Exe: []string{"a"}, GPU: "gpu1", Action: ActionTrack},
} {
if err := r.Compile(); err == nil {
t.Errorf("%+v compiled", r)
@@ -192,6 +195,50 @@ func TestWatcherHoldsEveryCard(t *testing.T) {
}
}
+// A hold whose rule names gpu1 asks gpu1 alone, and only gpu1 reports the
+// claim active; a rule without cards still holds every card.
+func TestWatcherHoldsTheCardsItNames(t *testing.T) {
+ cards := []*fakeBroker{{held: make(chan struct{}, 4)}, {held: make(chan struct{}, 4)}, {held: make(chan struct{}, 4)}}
+ w := &Watcher{Cards: []Broker{cards[0], cards[1], cards[2]}, Log: slog.New(slog.NewTextHandler(io.Discard, nil)),
+ Scan: func() ([]Proc, error) { return steam, nil }, holds: map[string]*hold{}}
+ w.Rules = compiled(t,
+ Rule{Name: "elden", Exe: []string{"eldenring.exe"}, GPU: "gpu1", Cards: []int{1}},
+ Rule{Name: "trainer", Exe: []string{"python3"}})
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ w.Tick(ctx, time.Now())
+ asked := func(i int) []string {
+ var out []string
+ for range 2 {
+ select {
+ case <-cards[i].held:
+ case <-time.After(200 * time.Millisecond):
+ }
+ }
+ cards[i].mu.Lock()
+ defer cards[i].mu.Unlock()
+ for _, r := range cards[i].asked {
+ out = append(out, r.Engine)
+ }
+ slices.Sort(out)
+ return out
+ }
+ for i, want := range []string{"claim:trainer", "claim:elden,claim:trainer", "claim:trainer"} {
+ if got := strings.Join(asked(i), ","); got != want {
+ t.Errorf("gpu%d asked for %s, want %s", i, got, want)
+ }
+ cards[i].mu.Lock()
+ if elden := cards[i].states[0]; elden.Name != "elden" || elden.Active != (i == 1) {
+ t.Errorf("gpu%d reports %+v, want elden active only on gpu1", i, elden)
+ }
+ if trainer := cards[i].states[1]; !trainer.Active {
+ t.Errorf("gpu%d reports %+v, want trainer active", i, trainer)
+ }
+ cards[i].mu.Unlock()
+ }
+}
+
// Windows-shaped process tables: Name is the exe, Cmdline the image path.
func TestBuiltinWindows(t *testing.T) {
ps := []Proc{
diff --git a/internal/claims/watch.go b/internal/claims/watch.go
index e695a18..bff7cd7 100644
--- a/internal/claims/watch.go
+++ b/internal/claims/watch.go
@@ -30,9 +30,9 @@ type Broker interface {
}
// Watcher scans the process table and keeps the claims of every card on the
-// node current. A hold holds every card: the process table cannot say which
-// card a game will draw on. Cards[0] notes the claim events, since the
-// node's cards share one event log.
+// node current. A hold holds the cards its rule names, or every card: the
+// process table cannot say which card a game will draw on. Cards[0] notes
+// the claim events, since the node's cards share one event log.
type Watcher struct {
Rules []Rule // compiled
Every time.Duration
@@ -129,11 +129,24 @@ func (w *Watcher) Tick(ctx context.Context, now time.Time) {
}
states = append(states, st)
}
- for _, b := range w.Cards {
- b.SetClaims(tracked, states)
+ for i, b := range w.Cards {
+ b.SetClaims(tracked, w.onCard(states, i))
}
}
+// onCard is the claim states as card i reports them: a hold whose rule
+// names other cards is not active here. states holds one entry per rule,
+// in rule order.
+func (w *Watcher) onCard(states []proto.Claim, i int) []proto.Claim {
+ out := slices.Clone(states)
+ for j, r := range w.Rules {
+ if r.Cards != nil && !slices.Contains(r.Cards, i) {
+ out[j].Active, out[j].Lease = false, ""
+ }
+ }
+ return out
+}
+
// granted is the lease IDs the hold has been granted so far, comma
// separated. Called with w.mu held.
func (h *hold) granted() string {
@@ -146,11 +159,18 @@ func (w *Watcher) start(ctx context.Context, r *Rule, m []Proc, now time.Time) {
h := &hold{cancel: cancel, lastSeen: now, leases: make([]string, len(w.Cards))}
w.holds[r.Name] = h
who := names(m)
- w.Log.Info("claim started: holding the card", "claim", r.Name, "procs", who, "cards", len(w.Cards))
+ cards := r.Cards
+ if cards == nil {
+ for i := range w.Cards {
+ cards = append(cards, i)
+ }
+ }
+ w.Log.Info("claim started: holding the card", "claim", r.Name, "procs", who, "cards", len(cards))
w.Cards[0].Note(proto.Event{Kind: proto.EvClaimStarted, Engine: "claim:" + r.Name, Detail: who})
req := broker.Request{Engine: "claim:" + r.Name, Group: "claim", Priority: broker.Hold,
Needs: r.NeedsMiB << 20, Wait: HoldWait, Pinned: true}
- for i, b := range w.Cards {
+ for _, i := range cards {
+ b := w.Cards[i]
go func() {
l, err := b.Acquire(hctx, req, func(why string) {
w.mu.Lock()
diff --git a/internal/cli/claim_cards_test.go b/internal/cli/claim_cards_test.go
new file mode 100644
index 0000000..6e44487
--- /dev/null
+++ b/internal/cli/claim_cards_test.go
@@ -0,0 +1,45 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "bytes"
+ "log/slog"
+ "slices"
+ "strings"
+ "testing"
+
+ "github.com/T-Crypt/berth/internal/claims"
+ "github.com/T-Crypt/berth/internal/node"
+)
+
+// A claim's gpu resolves to the node's card positions by name or PCI
+// address; a card the node lacks is logged and the claim holds every card.
+func TestClaimCardsResolvesTheRulesCards(t *testing.T) {
+ n, err := node.New(nil, nil,
+ &node.Card{Name: "gpu0", Addr: "0000:01:00.0"},
+ &node.Card{Name: "gpu1", Addr: "0000:41:00.0"},
+ &node.Card{Name: "gpu2", Addr: "0000:81:00.0"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ rules := []claims.Rule{
+ {Name: "all"},
+ {Name: "named", GPU: "gpu2,00000000:41:00.0"},
+ {Name: "missing", GPU: "gpu7"},
+ }
+ var logged bytes.Buffer
+ claimCards(rules, n, slog.New(slog.NewTextHandler(&logged, nil)))
+ for _, tc := range []struct {
+ rule int
+ want []int
+ }{{0, nil}, {1, []int{2, 1}}, {2, nil}} {
+ if got := rules[tc.rule].Cards; !slices.Equal(got, tc.want) {
+ t.Errorf("%s: cards = %v, want %v", rules[tc.rule].Name, got, tc.want)
+ }
+ }
+ if !strings.Contains(logged.String(), "claim=missing") || !strings.Contains(logged.String(), "holds every card") {
+ t.Errorf("the missing card was not logged: %q", logged.String())
+ }
+}
diff --git a/internal/cli/commands.go b/internal/cli/commands.go
index 2f0acb5..fc217e1 100644
--- a/internal/cli/commands.go
+++ b/internal/cli/commands.go
@@ -240,11 +240,13 @@ device for the actions.`,
{
Name: "hold",
Summary: "keep the card free for a game or a call; engines wait",
- Args: "[--for D] [--reason R] [--needs-mib N] [--wait D]",
+ Args: "[--gpu CARD] [--for D] [--reason R] [--needs-mib N] [--wait D]",
Help: `Hold the card for yourself. Every engine on it is drained the usual way,
none starts while the hold lasts, and nothing drains the hold: it is a pinned
lease at the top priority. It ends on Ctrl-C, after --for, or when this
process dies, so a forgotten terminal never strands the card for long.
+--gpu names the card on a node with several (gpu1, its PCI address, or a comma
+list); without it the hold takes the node's first card.
Games you launch from Steam, Lutris or gamescope are held for automatically
by the built-in claims (see berth claims); berth hold is for everything else.`,
@@ -518,6 +520,28 @@ unpairs every device.`,
Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
Run: runPair,
},
+ {
+ Name: "cordon",
+ Summary: "stop new grants on a card; what holds it keeps running",
+ Args: "CARD [--socket P] [--format F]",
+ Help: `Stop new grants on the named card, by its name (gpu1) or PCI address, for
+maintenance or before a VM takes it, while the node's other cards keep
+serving. Waiters stay in line with a reason that names the cordon, and
+the gateway sends a plain /v1 request to another card that serves its
+model. A hold (berth hold, a game claim) is still granted. What holds the
+card keeps running: berth stop ENGINE drains it. The cordon lasts until
+berth uncordon or until berthd restarts.`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runCordon,
+ },
+ {
+ Name: "uncordon",
+ Summary: "let new work get a cordoned card again",
+ Args: "CARD [--socket P] [--format F]",
+ Help: `Lift berth cordon from the named card. Its waiters are scheduled at once.`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runUncordon,
+ },
{
Name: "stop",
Summary: "stop an engine on purpose: its leases drain through their own wrappers",
@@ -605,11 +629,13 @@ docs/vfio.md has the requirements and an example script.`,
{
Name: "run",
Summary: "hold the card, run the switch script, wait for the card back",
- Args: "[--reason NAME] [--wait D] [--before CMD] [--after CMD] -- CMD [ARGS...]",
+ Args: "[--gpu CARD] [--reason NAME] [--wait D] [--before CMD] [--after CMD] -- CMD [ARGS...]",
Help: `Acquire the VFIO hold the way berth hold does: engines drain at their
boundaries and none starts. Then the script runs with the terminal attached, so
its sudo prompt reaches you, and with BERTH_VFIO_DEVICES set to the card's PCI
-functions, so it hardcodes no address.
+functions, so it hardcodes no address. --gpu names the card on a node with
+several (gpu1 or its PCI address); without it the run takes the node's first
+card.
--before runs once the engines are off the card: stop your own services that
hold it there. --after runs once the hold has ended, whenever --before ran,
diff --git a/internal/cli/cordon.go b/internal/cli/cordon.go
new file mode 100644
index 0000000..69cd1e8
--- /dev/null
+++ b/internal/cli/cordon.go
@@ -0,0 +1,68 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+// berth cordon and berth uncordon: a person stops new grants on a card,
+// for maintenance or before a VM takes it, while the other cards serve.
+
+import (
+ "strings"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+func runCordon(c *Context) int { return cordonCmd(c, proto.OpCordon) }
+func runUncordon(c *Context) int { return cordonCmd(c, proto.OpUncordon) }
+
+func cordonCmd(c *Context, op string) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ if len(c.Args) != 1 {
+ return c.fail(Failf("berth %s takes one card", op).WithHint("usage: berth %s CARD (berth status names the cards)", op))
+ }
+ conn, raw, err := server.Dial(*sock)
+ if err != nil {
+ return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: op, Card: c.Args[0]}); err != nil {
+ return c.fail(Systemf("berthd on %s: %v", *sock, err))
+ }
+ m, err := conn.Recv()
+ if err != nil {
+ return c.fail(Systemf("no %s reply from berthd: %v", op, err))
+ }
+ if m.Op == proto.OpRefused {
+ return c.fail(Failf("%s", m.Reason))
+ }
+ return c.Write(cordonView{op: op, cards: m.Cards}, nil)
+}
+
+type cordonView struct {
+ op string
+ cards []string
+}
+
+func (v cordonView) Data() any {
+ return struct {
+ Cordoned bool `json:"cordoned"`
+ Cards []string `json:"cards"`
+ }{v.op == proto.OpCordon, v.cards}
+}
+
+func (v cordonView) Human(c *Context, ok bool) error {
+ names := strings.Join(v.cards, ", ")
+ if v.op == proto.OpUncordon {
+ c.Printf("%s uncordoned: new work gets the card again\n", names)
+ return nil
+ }
+ c.Printf("%s cordoned: no new grants but holds\n", names)
+ c.Bullet("what holds it keeps running; berth stop ENGINE drains it")
+ c.Bullet("the cordon lasts until berth uncordon or a berthd restart")
+ return nil
+}
diff --git a/internal/cli/cordon_test.go b/internal/cli/cordon_test.go
new file mode 100644
index 0000000..bc723d7
--- /dev/null
+++ b/internal/cli/cordon_test.go
@@ -0,0 +1,50 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// berth cordon and uncordon against a live daemon: the report names the
+// card, status shows the cordon, and an unknown card fails with exit 1.
+func TestCordonAgainstADaemon(t *testing.T) {
+ sock, _, _ := startStopDaemon(t, time.Second)
+ cordoned := func() bool {
+ code, out, _ := runCLI(t, "status", "--socket", sock, "--format", "json")
+ if code != 0 {
+ t.Fatalf("status exit %d", code)
+ }
+ var st proto.Status
+ if err := json.Unmarshal([]byte(out), &st); err != nil {
+ t.Fatal(err)
+ }
+ return len(st.GPUs) > 0 && st.GPUs[0].Cordoned
+ }
+
+ code, out, errOut := runCLI(t, "cordon", "gpu0", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "gpu0") {
+ t.Fatalf("cordon: exit %d, out %q, err %q", code, out, errOut)
+ }
+ if !cordoned() {
+ t.Error("status does not show gpu0 cordoned")
+ }
+ if _, out, _ := runCLI(t, "status", "--socket", sock); !strings.Contains(out, "CORDONED") {
+ t.Errorf("berth status does not say CORDONED:\n%s", out)
+ }
+ if code, out, errOut = runCLI(t, "uncordon", "gpu0", "--socket", sock); code != 0 || !strings.Contains(out, "gpu0") {
+ t.Fatalf("uncordon: exit %d, out %q, err %q", code, out, errOut)
+ }
+ if cordoned() {
+ t.Error("status still shows gpu0 cordoned")
+ }
+ if code, _, errOut = runCLI(t, "cordon", "gpu7", "--socket", sock); code != ExitFail || !strings.Contains(errOut, "no card gpu7") {
+ t.Errorf("unknown card: exit %d, err %q, want exit %d naming the card", code, errOut, ExitFail)
+ }
+}
diff --git a/internal/cli/daemon.go b/internal/cli/daemon.go
index 1ef48e2..16188dc 100644
--- a/internal/cli/daemon.go
+++ b/internal/cli/daemon.go
@@ -10,6 +10,7 @@ import (
"os"
"os/signal"
"path/filepath"
+ "slices"
"strings"
"sync/atomic"
"syscall"
@@ -236,6 +237,7 @@ func runDaemon(c *Context) int {
// Claims: games and apps Berth does not run hold the card or have their
// memory named, from the process table. Validate already compiled them.
if rules, err := cfg.ClaimRules(); err == nil && len(rules) > 0 {
+ claimCards(rules, n, log)
cw := &claims.Watcher{Rules: rules, Every: cfg.Claims.Every.Duration, Cards: brokers, Log: log}
go cw.Run(ctx)
hold, track := 0, 0
@@ -464,3 +466,23 @@ func parkMaxBytes(cfg config.Config) uint64 {
}
return uint64(cfg.Park.MaxGiB * float64(gpu.GiB))
}
+
+// claimCards resolves each hold rule's gpu to positions in the node's cards.
+// A gpu the node does not have is logged, and that rule holds every card: a
+// game that is running is better held on too many cards than on none.
+func claimCards(rules []claims.Rule, n *node.Node, log *slog.Logger) {
+ for i := range rules {
+ r := &rules[i]
+ if r.GPU == "" {
+ continue
+ }
+ picked, err := n.Pick(r.GPU)
+ if err != nil {
+ log.Error("claim names a card this node does not have; it holds every card", "claim", r.Name, "err", err)
+ continue
+ }
+ for _, c := range picked {
+ r.Cards = append(r.Cards, slices.Index(n.Cards(), c))
+ }
+ }
+}
diff --git a/internal/cli/events_test.go b/internal/cli/events_test.go
index ae61c1c..509e114 100644
--- a/internal/cli/events_test.go
+++ b/internal/cli/events_test.go
@@ -15,6 +15,7 @@ import (
"github.com/T-Crypt/berth/internal/broker"
"github.com/T-Crypt/berth/internal/eventlog"
+ "github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/procs"
"github.com/T-Crypt/berth/internal/proto"
@@ -24,6 +25,15 @@ import (
// startNode runs a real berthd over one broker per named card, sharing one
// event log as berthd's cards do, and notes one event on the last card.
func startNode(t *testing.T, names ...string) string {
+ t.Helper()
+ sock, brokers := serveNode(t, names...)
+ brokers[len(brokers)-1].Note(proto.Event{Kind: proto.EvAlertRaised, Detail: "too hot"})
+ return sock
+}
+
+// serveNode runs a real berthd over one broker per named card, the card at
+// position i reading as nvidia-smi index i, and returns the brokers in order.
+func serveNode(t *testing.T, names ...string) (string, []*broker.Broker) {
t.Helper()
sock := filepath.Join(t.TempDir(), "berth.sock")
log := slog.New(slog.NewTextHandler(io.Discard, nil))
@@ -31,27 +41,44 @@ func startNode(t *testing.T, names ...string) string {
t.Cleanup(cancel)
events, ids := eventlog.New(), &atomic.Uint64{}
var cards []*node.Card
- for _, name := range names {
+ var brokers []*broker.Broker
+ for i, name := range names {
b := broker.New(broker.Config{
Card: name, Events: events, IDs: ids,
Poll: 20 * time.Millisecond, DefaultWait: 5 * time.Second, JuggleWindow: -1,
Version: "test", Log: log,
- }, emptyCard{}, procs.System{})
+ }, indexCard(i), procs.System{})
b.Tick(ctx)
go b.Run(ctx)
cards = append(cards, &node.Card{Name: name, B: b})
+ brokers = append(brokers, b)
}
n, err := node.New(events, ids, cards...)
if err != nil {
t.Fatal(err)
}
- cards[len(cards)-1].B.Note(proto.Event{Kind: proto.EvAlertRaised, Detail: "too hot"})
ln, err := server.Listen(sock)
if err != nil {
t.Fatal(err)
}
go server.Serve(ctx, ln, n, server.Extras{}, log)
- return sock
+ return sock, brokers
+}
+
+// indexCard is an empty card that nvidia-smi numbers with its value.
+type indexCard int
+
+func (i indexCard) Read(context.Context) (gpu.Snapshot, error) {
+ return gpu.Snapshot{Index: int(i), Total: 24 << 30, Free: 24 << 30, At: time.Now()}, nil
+}
+
+// leaseNames is the engines holding a lease on b.
+func leaseNames(b *broker.Broker) []string {
+ var out []string
+ for _, l := range b.Status().Leases {
+ out = append(out, l.Engine)
+ }
+ return out
}
// An event line names its card once berthd runs more than one, as the CARD
diff --git a/internal/cli/helpers_test.go b/internal/cli/helpers_test.go
index a322ff5..5444d58 100644
--- a/internal/cli/helpers_test.go
+++ b/internal/cli/helpers_test.go
@@ -10,6 +10,7 @@ import (
"path/filepath"
"strings"
"testing"
+ "time"
)
// isolate points every path the CLI reads at a temp dir, so a test never
@@ -113,3 +114,15 @@ func boolInt(b bool) int {
}
return 0
}
+
+// waitUntil polls until f is true, so async daemon-side releases settle.
+func waitUntil(t *testing.T, f func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(5 * time.Second)
+ for !f() {
+ if time.Now().After(deadline) {
+ t.Fatal("the condition never became true")
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+}
diff --git a/internal/cli/hold.go b/internal/cli/hold.go
index 0ed314d..c4e3c9c 100644
--- a/internal/cli/hold.go
+++ b/internal/cli/hold.go
@@ -28,6 +28,7 @@ func runHold(c *Context) int {
reason := fs.String("reason", "", "shown on the lease, e.g. game or call (default: hold)")
dur := fs.Duration("for", 0, "end the hold after this long; 0 = until Ctrl-C")
needs := fs.Uint64("needs-mib", 0, "hold only this much memory instead of the whole card")
+ card := fs.String("gpu", "", "card to hold: its name on the node (gpu1) or its PCI address, or a comma list of them for one hold on several cards; default the node's first card")
wait := fs.Duration("wait", 10*time.Minute, "how long to wait for engines to drain; 0 = berthd's default; negative = no deadline")
if code, done := c.Parse(fs); done {
return code
@@ -35,6 +36,11 @@ func runHold(c *Context) int {
if code, done := c.noArgs(); done {
return code
}
+ if *card != "" {
+ if err := gpu.CheckCard(*card); err != nil {
+ return c.fail(Systemf("%v", err))
+ }
+ }
name := "hold"
if r := strings.TrimSpace(*reason); r != "" {
name = "hold:" + strings.ReplaceAll(r, " ", "-")
@@ -49,7 +55,7 @@ func runHold(c *Context) int {
}
defer raw.Close()
go func() { <-ctx.Done(); raw.Close() }()
- if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Engine: name, Group: "hold", Priority: "hold",
+ if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Engine: name, Card: *card, Group: "hold", Priority: "hold",
Needs: *needs * gpu.MiB, Wait: wait.Seconds(), Pinned: true}); err != nil {
return c.fail(Systemf("berthd on %s: %v", *sock, err))
}
@@ -78,7 +84,11 @@ func runHold(c *Context) int {
until = "for " + dur.String()
timer = time.After(*dur)
}
- fmt.Fprintf(c.Err, "berth: card held as %s (%s); engines wait until it ends\n", m.Lease, until)
+ held := "card"
+ if *card != "" {
+ held = *card
+ }
+ fmt.Fprintf(c.Err, "berth: %s held as %s (%s); engines wait until it ends\n", held, m.Lease, until)
if m.Reason != "" {
fmt.Fprintf(c.Err, "berth: note: %s\n", m.Reason)
}
diff --git a/internal/cli/hold_test.go b/internal/cli/hold_test.go
new file mode 100644
index 0000000..2aa1d78
--- /dev/null
+++ b/internal/cli/hold_test.go
@@ -0,0 +1,76 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "slices"
+ "strings"
+ "testing"
+)
+
+// berth hold --gpu holds the card it names on a node of several, and leaves
+// the others to their engines.
+func TestHoldTakesTheCardItNames(t *testing.T) {
+ isolate(t)
+ for _, tc := range []struct {
+ gpu string
+ want []bool // held, per card
+ }{
+ {"gpu1", []bool{false, true, false}},
+ {"gpu0,gpu2", []bool{true, false, true}},
+ {"", []bool{true, false, false}},
+ } {
+ sock, brokers := serveNode(t, "gpu0", "gpu1", "gpu2")
+ args := []string{"hold", "--socket", sock, "--for", "1s", "--reason", "game"}
+ if tc.gpu != "" {
+ args = append(args, "--gpu", tc.gpu)
+ }
+ done := make(chan string, 1)
+ go func() { _, _, errOut := runCLI(t, args...); done <- errOut }()
+ held := func(b int) bool { return slices.Contains(leaseNames(brokers[b]), "hold:game") }
+ waitUntil(t, func() bool {
+ for i, want := range tc.want {
+ if want && !held(i) {
+ return false
+ }
+ }
+ return true
+ })
+ for i, want := range tc.want {
+ if !want && held(i) {
+ t.Errorf("--gpu %q: gpu%d is held too", tc.gpu, i)
+ }
+ }
+ if errOut := <-done; !strings.Contains(errOut, "hold released") {
+ t.Errorf("--gpu %q: the hold did not end cleanly: %q", tc.gpu, errOut)
+ }
+ }
+}
+
+// berth hold refuses a --gpu no card can be, before berthd is asked.
+func TestHoldRefusesABadGPU(t *testing.T) {
+ isolate(t)
+ sock, brokers := serveNode(t, "gpu0", "gpu1")
+ for _, bad := range []string{"card one", "gpu1,gpu1"} {
+ code, _, errOut := runCLI(t, "hold", "--socket", sock, "--gpu", bad)
+ if code == ExitOK || !strings.Contains(errOut, "gpu") {
+ t.Errorf("--gpu %q: exit = %d, stderr %q; want a refusal naming the card", bad, code, errOut)
+ }
+ }
+ for i, b := range brokers {
+ if n := leaseNames(b); len(n) > 0 {
+ t.Errorf("gpu%d holds %v after a refused hold", i, n)
+ }
+ }
+}
+
+// berth hold --gpu names a card berthd does not have: refused, nothing held.
+func TestHoldRefusesACardTheNodeLacks(t *testing.T) {
+ isolate(t)
+ sock, _ := serveNode(t, "gpu0", "gpu1")
+ code, _, errOut := runCLI(t, "hold", "--socket", sock, "--gpu", "gpu7", "--wait", "1s", "--for", "1s")
+ if code == ExitOK || !strings.Contains(errOut, "no card gpu7") {
+ t.Errorf("exit = %d, stderr %q; want a refusal that names gpu7", code, errOut)
+ }
+}
diff --git a/internal/cli/run.go b/internal/cli/run.go
index 9ebff35..43f2a75 100644
--- a/internal/cli/run.go
+++ b/internal/cli/run.go
@@ -17,6 +17,7 @@ import (
"github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/runner"
"github.com/T-Crypt/berth/internal/suite"
+ "github.com/T-Crypt/berth/internal/sysd"
)
func runRun(c *Context) int {
@@ -44,6 +45,7 @@ func runRun(c *Context) int {
container := fs.String("container", "", "NAME of the one container CMD runs in: the drain stops it via docker/podman stop instead of signalling the wrapper")
card := fs.String("gpu", "", "card to run on: its name on the node (gpu1) or its PCI address, or a comma list of them for one lease on several cards; the lease goes there and the engine gets CUDA_VISIBLE_DEVICES set to the cards' UUIDs unless its environment sets it; over the entry's gpu; default the node's first card")
composeFile := fs.String("compose", "", "absolute PATH of the Compose file CMD's project runs from: the lease covers every container of the project and the drain stops it with compose stop, never down; exclusive with --container")
+ sliceName := fs.String("slice", "", "NAME.slice to start the engine's scope under (systemd user manager, Linux): the slice unit's CPU, memory and swap policy applies to the engine and everything it forks; over the entry's slice")
grace := fs.Duration("grace", 30*time.Second, "time each drain step gets before the next")
failClosed := fs.Bool("fail-closed", false, "do not start CMD without berthd or stop it without a safe boundary")
cfgPath := fs.String("config", config.DefaultPath(), "berth.toml: the engine entry matching --as (or --group) supplies the settings a flag does not set; a set flag wins and says so; missing = no defaults ($BERTH_CONFIG)")
@@ -87,6 +89,17 @@ func runRun(c *Context) int {
if c.Changed("api-key-file") {
e.APIKeyFile = *keyFile
}
+ if c.Changed("slice") {
+ if e.Slice != "" && e.Slice != *sliceName {
+ fmt.Fprintf(c.Err, "berth: --slice overrides berth.toml slice = %s for %s\n", e.Slice, e.Name)
+ }
+ e.Slice = *sliceName
+ }
+ if e.Slice != "" {
+ if err := sysd.CheckSlice(e.Slice); err != nil {
+ return c.fail(Systemf("%v", err))
+ }
+ }
if e.APIKeyEnv != "" && e.APIKeyFile != "" {
return c.fail(Systemf("api_key_env and api_key_file are mutually exclusive"))
}
diff --git a/internal/cli/run_test.go b/internal/cli/run_test.go
index 385f674..8a5dba8 100644
--- a/internal/cli/run_test.go
+++ b/internal/cli/run_test.go
@@ -72,6 +72,19 @@ func TestRunRejectsABadPriority(t *testing.T) {
}
}
+// A slice that cannot name a unit fails at the CLI, before berthd is
+// dialed and the engine is touched.
+func TestRunRejectsABadSlice(t *testing.T) {
+ isolate(t)
+ code, _, errOut := runCLI(t, "run", "--as", "x", "--slice", "nope", "--", "true")
+ if code != ExitSystem {
+ t.Fatalf("exit = %d, want %d", code, ExitSystem)
+ }
+ if !strings.Contains(errOut, "must name a .slice unit") {
+ t.Errorf("stderr = %q", errOut)
+ }
+}
+
// --fail-closed with berthd down refuses to start the engine: EX_TEMPFAIL, so
// a service manager can retry instead of running unprotected.
func TestRunFailClosedRefusesWithoutADaemon(t *testing.T) {
diff --git a/internal/cli/status.go b/internal/cli/status.go
index 40bc5cb..7a1aa3b 100644
--- a/internal/cli/status.go
+++ b/internal/cli/status.go
@@ -51,6 +51,9 @@ func (v statusView) Human(c *Context, ok bool) error {
role := "inventory only"
if g.Scheduled {
role = "leased by berth, baseline " + server.MiB(g.Baseline)
+ if g.Cordoned {
+ role = c.Style.Err("CORDONED") + ", holds only; " + role
+ }
}
c.Printf("gpu%d %s: %s used of %s, %s free (%s, read %s ago)\n", g.Index, g.Name,
server.MiB(g.Used), server.MiB(g.Total), server.MiB(g.Free), role, age(g.Age))
diff --git a/internal/cli/vfio.go b/internal/cli/vfio.go
index 709d70a..3e2a7fc 100644
--- a/internal/cli/vfio.go
+++ b/internal/cli/vfio.go
@@ -12,6 +12,7 @@ import (
"text/tabwriter"
"time"
+ "github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/server"
"github.com/T-Crypt/berth/internal/vfio"
@@ -32,10 +33,17 @@ type vfioCard struct {
// readable while the card is bound to vfio-pci and nvidia-smi cannot see it;
// berthd is asked either way. Otherwise the status berthd reports names the scheduled
// index, and nvidia-smi names that card's bus id; when berthd is down or
-// has no reading yet, index 0.
-func vfioCardAt(sock, explicit, smiBin string) (vfioCard, *Error) {
+// has no reading yet, index 0. key is the card a run names (vfio run
+// --gpu), "" for berthd's first card; berthd's status is narrowed to it.
+func vfioCardAt(sock, explicit, key, smiBin string) (vfioCard, *Error) {
var card vfioCard
st, err := vfioStatusOf(sock)
+ if err == nil && key != "" {
+ if st = node.View(st, key); st == nil {
+ return card, Failf("berthd has no card %s", key).
+ WithHint("berth status names the cards berthd schedules")
+ }
+ }
card.Daemon, card.DialErr = st, err
if explicit != "" {
card.Addr = vfio.SysAddr(explicit)
@@ -301,7 +309,7 @@ func vfioStatus(c *Context, r vfio.Reader, smiBin string) int {
if len(c.Args) == 1 {
explicit = c.Args[0]
}
- card, e := vfioCardAt(*sock, explicit, smiBin)
+ card, e := vfioCardAt(*sock, explicit, "", smiBin)
if e != nil {
return c.fail(e)
}
diff --git a/internal/cli/vfio_run.go b/internal/cli/vfio_run.go
index a696b4c..90b3c3f 100644
--- a/internal/cli/vfio_run.go
+++ b/internal/cli/vfio_run.go
@@ -12,6 +12,7 @@ import (
"syscall"
"time"
+ "github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/runner"
"github.com/T-Crypt/berth/internal/server"
@@ -114,6 +115,7 @@ func vfioRun(c *Context, o vfioRunOpts) int {
wait := fs.Duration("wait", 10*time.Minute, "how long to wait for engines to drain; 0 = berthd's default; negative = no deadline")
before := fs.String("before", "", "shell command run once the engines are off the card, before the check that nothing holds it (stop your own services here)")
after := fs.String("after", "", "shell command run once the hold has ended, whenever --before ran (start those services again)")
+ gpuKey := fs.String("gpu", "", "card to pass through: its name on the node (gpu1) or its PCI address; default the node's first card")
if code, done := c.Parse(fs); done {
return code
}
@@ -122,9 +124,17 @@ func vfioRun(c *Context, o vfioRunOpts) int {
WithHint("usage: berth vfio run [--reason NAME] -- CMD [ARGS...]"))
}
argv := c.Args
+ if *gpuKey != "" {
+ if err := gpu.CheckCard(*gpuKey); err != nil {
+ return c.fail(Systemf("%v", err))
+ }
+ if strings.Contains(*gpuKey, ",") {
+ return c.fail(Systemf("--gpu %s names several cards; a passthrough moves one card", *gpuKey))
+ }
+ }
// The card, by way of berthd's scheduled index and nvidia-smi.
- card, e := vfioCardAt(*sock, "", o.smiBin)
+ card, e := vfioCardAt(*sock, "", *gpuKey, o.smiBin)
if e != nil {
return c.fail(e)
}
@@ -176,7 +186,7 @@ func vfioRun(c *Context, o vfioRunOpts) int {
raw.Close()
}
}()
- if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Engine: name, Group: proto.GroupVFIO, Priority: "hold",
+ if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Engine: name, Card: *gpuKey, Group: proto.GroupVFIO, Priority: "hold",
Wait: wait.Seconds(), Pinned: true}); err != nil {
return c.fail(Systemf("berthd on %s: %v", *sock, err))
}
diff --git a/internal/cli/vfio_test.go b/internal/cli/vfio_test.go
index 2a11ea6..9d049a2 100644
--- a/internal/cli/vfio_test.go
+++ b/internal/cli/vfio_test.go
@@ -215,18 +215,6 @@ func startVfioDaemon(t *testing.T) (string, *broker.Broker) {
return sock, b
}
-// waitUntil polls until f is true, so async daemon-side releases settle.
-func waitUntil(t *testing.T, f func() bool) {
- t.Helper()
- deadline := time.Now().Add(5 * time.Second)
- for !f() {
- if time.Now().After(deadline) {
- t.Fatal("the condition never became true")
- }
- time.Sleep(5 * time.Millisecond)
- }
-}
-
// none is true when the daemon holds no lease.
func none(b *broker.Broker) func() bool {
return func() bool { return len(b.Status().Leases) == 0 }
@@ -670,3 +658,72 @@ func TestVfioStatusNamesTheVMNotAStaleReading(t *testing.T) {
t.Errorf("berthd's line = %q, want the VM state and no current-looking memory figure", got)
}
}
+
+// TestVfioRunTakesTheCardItNames: on a node of two cards, --gpu gpu1 holds
+// gpu1 alone and passes through the card at nvidia-smi index 1. Index 0 is
+// another card with no functions in the tree, so a run that resolved the
+// first card would fail its preflight.
+func TestVfioRunTakesTheCardItNames(t *testing.T) {
+ isolate(t)
+ sock, brokers := serveNode(t, "gpu0", "gpu1")
+ tr := newVfioTree(t)
+ tr.function(vfioAddr, "nvidia")
+ tr.group()
+ smi := tr.script("nvidia-smi", "case \" $* \" in *\" -i 0 \"*) echo 00000000:02:00.0; exit 0;; esac\nexec "+tr.smi()+" \"$@\"\n")
+ held, resume := filepath.Join(t.TempDir(), "held"), filepath.Join(t.TempDir(), "resume")
+ cmd := tr.script("switch", fmt.Sprintf("touch %s\nwhile [ ! -e %s ]; do sleep 0.01; done\n%s%sexit 0\n",
+ held, resume, tr.flip("vfio-pci"), tr.flip("nvidia")))
+ done := make(chan string, 1)
+ go func() {
+ code, _, errOut := runVfioRunIn(t, tr, smi, vfioRunOpts{settle: time.Millisecond, poll: 5 * time.Millisecond},
+ "--socket", sock, "--gpu", "gpu1", "--", cmd)
+ done <- fmt.Sprintf("exit %d: %s", code, errOut)
+ }()
+ waitUntil(t, func() bool {
+ select {
+ case out := <-done:
+ t.Fatalf("the run ended before its script held the card: %s", out)
+ default:
+ }
+ _, err := os.Stat(held)
+ return err == nil
+ })
+ if got := leaseNames(brokers[1]); len(got) != 1 || got[0] != "vfio" {
+ t.Errorf("gpu1 leases = %v, want the vfio hold", got)
+ }
+ if got := leaseNames(brokers[0]); len(got) != 0 {
+ t.Errorf("gpu0 leases = %v, want none", got)
+ }
+ if err := os.WriteFile(resume, nil, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if out := <-done; !strings.HasPrefix(out, "exit 0:") || !strings.Contains(out, "the card is back; hold released") {
+ t.Errorf("run = %q, want exit 0 with the card back", out)
+ }
+ waitUntil(t, none(brokers[1]))
+}
+
+// TestVfioRunRefusesSeveralCardsOrOneTheNodeLacks: a passthrough moves one
+// card berthd schedules; anything else is refused before a hold is asked.
+func TestVfioRunRefusesSeveralCardsOrOneTheNodeLacks(t *testing.T) {
+ isolate(t)
+ sock, brokers := serveNode(t, "gpu0", "gpu1")
+ tr := newVfioTree(t)
+ tr.function(vfioAddr, "nvidia")
+ tr.group()
+ for _, tc := range []struct{ gpu, want string }{
+ {"gpu0,gpu1", "a passthrough moves one card"},
+ {"gpu7", "berthd has no card gpu7"},
+ } {
+ code, _, errOut := runVfioRunIn(t, tr, tr.smi(), vfioRunOpts{settle: time.Millisecond},
+ "--socket", sock, "--gpu", tc.gpu, "--", "true")
+ if code == ExitOK || !strings.Contains(errOut, tc.want) {
+ t.Errorf("--gpu %s: exit = %d, stderr %q; want %q", tc.gpu, code, errOut, tc.want)
+ }
+ }
+ for i, b := range brokers {
+ if n := leaseNames(b); len(n) > 0 {
+ t.Errorf("gpu%d holds %v after a refused run", i, n)
+ }
+ }
+}
diff --git a/internal/config/config.go b/internal/config/config.go
index 6a4c690..7f7da97 100644
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -26,6 +26,7 @@ import (
"github.com/T-Crypt/berth/internal/notify"
"github.com/T-Crypt/berth/internal/park"
"github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/sysd"
)
// Engine is one inference engine this node reads metrics from. Reading is
@@ -129,6 +130,13 @@ type Engine struct {
// sets CUDA_VISIBLE_DEVICES to their UUIDs unless the engine's
// environment sets it. "" = the node's first card.
GPU string `toml:"gpu,omitempty" json:"gpu,omitempty" yaml:"gpu,omitempty"`
+ // Slice names a systemd slice (NAME.slice) to place the engine's
+ // transient scope under, so the slice unit's resource policy, CPU,
+ // memory and swap limits, applies to the engine and everything it
+ // forks, not only to the wrapper. systemd user manager, Linux;
+ // elsewhere the key is accepted and unused. berth run --slice wins
+ // over it.
+ Slice string `toml:"slice,omitempty" json:"slice,omitempty" yaml:"slice,omitempty"`
// Park picks the RAM tier on a drain: off (the default) runs the
// plain drain, hook asks the engine to move its weights to system
// RAM instead, so the next use resumes in seconds. Engines with a
@@ -278,6 +286,9 @@ type ClaimRule struct {
Action string `toml:"action,omitempty" json:"action,omitempty" yaml:"action,omitempty"`
NeedsMiB uint64 `toml:"needs_mib,omitzero" json:"needs_mib,omitempty" yaml:"needs_mib,omitempty"`
Linger Duration `toml:"linger,omitempty" json:"linger,omitempty" yaml:"linger,omitempty"`
+ // GPU is the cards a hold takes on a node with several, as berth run
+ // --gpu names them; "" is every card.
+ GPU string `toml:"gpu,omitempty" json:"gpu,omitempty" yaml:"gpu,omitempty"`
}
// ClaimRules is the rule list berthd runs: the configured rules first (a
@@ -288,7 +299,7 @@ func (c *Config) ClaimRules() ([]claims.Rule, error) {
names := map[string]bool{}
for _, r := range c.Claim {
rule := claims.Rule{Name: r.Name, Exe: r.Exe, Cmdline: r.Cmdline, Not: r.Not, Parent: r.Parent, ParentCmdline: r.ParentCmdline,
- Action: r.Action, NeedsMiB: r.NeedsMiB, Linger: r.Linger.Duration}
+ Action: r.Action, NeedsMiB: r.NeedsMiB, Linger: r.Linger.Duration, GPU: r.GPU}
if err := rule.Compile(); err != nil {
return nil, err
}
@@ -610,7 +621,7 @@ func (c *Config) Validate() error {
ordered[o] = true
}
seen := map[string]bool{}
- served := map[string]string{} // model -> the engine that serves it
+ served := map[string][]*Engine{} // model -> the engines that serve it
for i, e := range c.Engine {
where := fmt.Sprintf("engine[%d] %q", i, e.Name)
if e.Name == "" {
@@ -651,11 +662,13 @@ func (c *Config) Validate() error {
errs = append(errs, fmt.Errorf("%s: a custom engine needs at least one of health, models, metrics, slots or [engine.map]", where))
}
for _, m := range e.Serves {
- if other, ok := served[m]; ok {
- errs = append(errs, fmt.Errorf("%s: serves %q, which %q already serves; the gateway would never pick this one", where, m, other))
- } else {
- served[m] = e.Name
+ // One model on several cards is one engine per card, which a
+ // per-card gateway URL picks between. Two on one card, or one
+ // that names no card, leave the gateway nothing to pick by.
+ if j := slices.IndexFunc(served[m], func(o *Engine) bool { return sharesCard(o.GPU, e.GPU) }); j >= 0 {
+ errs = append(errs, fmt.Errorf("%s: serves %q, which %q already serves; the gateway would never pick this one", where, m, served[m][j].Name))
}
+ served[m] = append(served[m], &c.Engine[i])
}
// A background engine starts when the card is free, not for a model.
if len(e.Start) > 0 && len(e.Serves) == 0 && e.Type != "comfyui" && !e.Background {
@@ -677,6 +690,11 @@ func (c *Config) Validate() error {
if e.KVSnapshots != "" && !filepath.IsAbs(e.KVSnapshots) {
errs = append(errs, fmt.Errorf("%s: kv_snapshots %q must be an absolute path: the gateway starts engines from its own working directory", where, e.KVSnapshots))
}
+ if e.Slice != "" {
+ if err := sysd.CheckSlice(e.Slice); err != nil {
+ errs = append(errs, fmt.Errorf("%s: %w", where, err))
+ }
+ }
if e.DrainMax.Duration < 0 {
errs = append(errs, fmt.Errorf("%s: drain_max %s must be >= 0", where, e.DrainMax))
}
@@ -856,3 +874,18 @@ func shelfValid(s string) bool {
}
return false
}
+
+// sharesCard is whether two engines' gpu keys may name one card: either
+// names no card (the first card), or their comma lists have a key in common.
+func sharesCard(a, b string) bool {
+ if a == "" || b == "" {
+ return true
+ }
+ bs := strings.Split(strings.ReplaceAll(b, " ", ""), ",")
+ for _, k := range strings.Split(strings.ReplaceAll(a, " ", ""), ",") {
+ if slices.Contains(bs, k) {
+ return true
+ }
+ }
+ return false
+}
diff --git a/internal/config/config_test.go b/internal/config/config_test.go
index b6e4cb9..c421d06 100644
--- a/internal/config/config_test.go
+++ b/internal/config/config_test.go
@@ -508,6 +508,37 @@ ttl = "-5s"
}
}
+func TestEngineSliceKeyValidated(t *testing.T) {
+ c, err := Load(write(t, `
+[[engine]]
+name = "strata"
+type = "llamacpp"
+url = "http://127.0.0.1:8080"
+slice = "strata.slice"
+[[engine]]
+name = "plain"
+type = "llamacpp"
+url = "http://127.0.0.1:8081"
+`))
+ if err != nil {
+ t.Fatalf("valid slice keys refused: %v", err)
+ }
+ if c.Engine[0].Slice != "strata.slice" || c.Engine[1].Slice != "" {
+ t.Fatalf("slice keys not loaded: %+v", c.Engine)
+ }
+ for _, bad := range []string{`slice = "strata"`, `slice = "a/b.slice"`, `slice = ".slice"`} {
+ _, err := Load(write(t, `
+[[engine]]
+name = "s"
+type = "llamacpp"
+url = "http://127.0.0.1:8080"
+`+bad+"\n"))
+ if err == nil || !strings.Contains(err.Error(), "slice") {
+ t.Errorf("%s: got %v, want a slice error", bad, err)
+ }
+ }
+}
+
func TestEngineParkKeysValidated(t *testing.T) {
c, err := Load(write(t, `
[[engine]]
@@ -997,3 +1028,47 @@ func TestSquattersAllow(t *testing.T) {
t.Errorf("allow = %q, want chrome,WindowsTerminal", got)
}
}
+
+func TestClaimGPUKeyValidated(t *testing.T) {
+ c, err := Load(write(t, `
+[[claim]]
+name = "sim"
+exe = ["sim.exe"]
+gpu = "gpu1,0000:41:00.0"
+`))
+ if err != nil {
+ t.Fatalf("valid claim gpu refused: %v", err)
+ }
+ rules, err := c.ClaimRules()
+ if err != nil || rules[0].Name != "sim" || rules[0].GPU != "gpu1,0000:41:00.0" {
+ t.Fatalf("claim gpu not carried to the rule: %+v, %v", rules, err)
+ }
+ for _, bad := range []struct{ toml, want string }{
+ {`gpu = "card 1"`, "must be a card name"},
+ {`gpu = "gpu1,gpu1"`, "names gpu1 twice"},
+ {"gpu = \"gpu1\"\naction = \"track\"", "a track claim holds none"},
+ } {
+ _, err := Load(write(t, "[[claim]]\nname = \"sim\"\nexe = [\"sim.exe\"]\n"+bad.toml+"\n"))
+ if err == nil || !strings.Contains(err.Error(), bad.want) {
+ t.Errorf("%s: got %v, want %q", bad.toml, err, bad.want)
+ }
+ }
+}
+
+// One model may be served on two cards, one engine on each, so a per-card
+// gateway URL can reach either. Two engines that serve it on the same card,
+// or where one names no card, still conflict.
+func TestOneModelOnTwoCards(t *testing.T) {
+ entry := func(name, gpu string, port int) string {
+ return fmt.Sprintf("[[engine]]\nname = %q\ntype = \"llamacpp\"\nurl = \"http://127.0.0.1:%d\"\ngpu = %q\nserves = [\"qwen\"]\n", name, port, gpu)
+ }
+ if _, err := Load(write(t, entry("a", "gpu0", 8080)+entry("b", "gpu1", 8081))); err != nil {
+ t.Errorf("qwen on gpu0 and gpu1: %v", err)
+ }
+ for _, pair := range [][2]string{{"gpu1", "gpu1"}, {"", "gpu1"}} {
+ _, err := Load(write(t, entry("a", pair[0], 8080)+entry("b", pair[1], 8081)))
+ if err == nil || !strings.Contains(err.Error(), `serves "qwen", which "a" already serves`) {
+ t.Errorf("qwen on %q and %q: %v, want the conflict", pair[0], pair[1], err)
+ }
+ }
+}
diff --git a/internal/engines/engines.go b/internal/engines/engines.go
index 42390ea..faff156 100644
--- a/internal/engines/engines.go
+++ b/internal/engines/engines.go
@@ -153,9 +153,9 @@ func (w *Watcher) Poll(ctx context.Context) {
w.mu.Unlock()
}
-// Finished records that the engine named name exited cleanly, so it
-// stopped answering on purpose: its latch clears, and a reading taken
-// before the exit cannot latch it again.
+// Finished records that the engine named name ended on purpose (exit 0,
+// or a stop its run carried out), so it stopped answering on purpose: its
+// latch clears, and a reading taken before the exit cannot latch it again.
func (w *Watcher) Finished(name string) {
if w == nil {
return
diff --git a/internal/gateway/background.go b/internal/gateway/background.go
index 12fc588..e65eebf 100644
--- a/internal/gateway/background.go
+++ b/internal/gateway/background.go
@@ -31,7 +31,7 @@ const DefaultIdleAfter = 60 * time.Second
// beside it is the collision Berth exists to prevent.
func cardFree(st *proto.Status) bool {
if st == nil || st.Passthrough != nil || st.ProbeErr != "" ||
- len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 {
+ len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 || cordoned(st) {
return false
}
// Status lists every claim rule; only one matching right now is in use.
diff --git a/internal/gateway/board.go b/internal/gateway/board.go
index 904a189..f63b99c 100644
--- a/internal/gateway/board.go
+++ b/internal/gateway/board.go
@@ -31,7 +31,11 @@ type boardCard struct {
Name string `json:"name"`
// State is free, held, draining (a holder is giving it back),
// passthrough (in a VM) or probe-failed.
- State string `json:"state"`
+ State string `json:"state"`
+ // Cordoned: berth cordon closed the card to new work but holds.
+ Cordoned bool `json:"cordoned,omitempty"`
+ // Base is the card's own gateway base URL, a path on this gateway.
+ Base string `json:"base,omitempty"`
Total uint64 `json:"total"`
Used uint64 `json:"used"`
Holders []guiHold `json:"holders"`
@@ -77,7 +81,11 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string
if !gp.Scheduled {
continue
}
- c := boardCard{Card: gp.Card, Index: gp.Index, Name: gp.Name, State: "free", Total: gp.Total, Used: gp.Used, Holders: []guiHold{}}
+ c := boardCard{Card: gp.Card, Index: gp.Index, Name: gp.Name, State: "free", Cordoned: gp.Cordoned,
+ Total: gp.Total, Used: gp.Used, Holders: []guiHold{}}
+ if gp.Card != "" { // a berthd older than card names has no per-card URL
+ c.Base = "/cards/" + gp.Card + "/v1"
+ }
for _, l := range node.View(status, gp.Card).Leases {
c.Holders = append(c.Holders, guiHold{Name: holderName(l), State: l.State, Pinned: l.Pinned})
if c.State == "free" {
diff --git a/internal/gateway/boardcards_test.go b/internal/gateway/boardcards_test.go
new file mode 100644
index 0000000..ee383e2
--- /dev/null
+++ b/internal/gateway/boardcards_test.go
@@ -0,0 +1,74 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "encoding/json"
+ "io"
+ "log/slog"
+ "net/http/httptest"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+)
+
+// On a node with two cards the board names each engine's card, so a phone
+// picks the card by the engine it launches, and gives each card its base
+// URL and its cordon. /gui/state carries the same cards per engine.
+func TestBoardNamesEachEnginesCard(t *testing.T) {
+ sock, n := twoCards(t)
+ if _, err := n.Cordon("gpu1", true); err != nil {
+ t.Fatal(err)
+ }
+ g := &Gateway{Socket: sock, Ready: 5 * time.Second, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil)),
+ Engines: []config.Engine{
+ {Name: "qwen-a", Type: "llamacpp", URL: "http://127.0.0.1:1", GPU: "gpu0", Serves: []string{"qwen"}},
+ {Name: "qwen-b", Type: "llamacpp", URL: "http://127.0.0.1:1", GPU: "0000:09:00.0", Serves: []string{"qwen"}},
+ {Name: "tts", Type: "llamacpp", URL: "http://127.0.0.1:1", Serves: []string{"voice"}},
+ }}
+ srv := httptest.NewServer(g.Handler())
+ defer srv.Close()
+
+ var b boardNode
+ until(t, "both cards on the board", func() bool { b = readBoard(t, srv.URL); return len(b.Cards) == 2 })
+ type card struct {
+ Card, Base string
+ Cordoned bool
+ }
+ var cards []card
+ for _, c := range b.Cards {
+ cards = append(cards, card{c.Card, c.Base, c.Cordoned})
+ }
+ if want := []card{{"gpu0", "/cards/gpu0/v1", false}, {"gpu1", "/cards/gpu1/v1", true}}; !reflect.DeepEqual(cards, want) {
+ t.Errorf("cards = %+v, want %+v", cards, want)
+ }
+ want := map[string][]string{"qwen-a": {"gpu0"}, "qwen-b": {"gpu1"}, "tts": {"gpu0"}}
+ got := map[string][]string{}
+ for _, e := range b.Engines {
+ got[e.Name] = e.Cards
+ }
+ if !reflect.DeepEqual(got, want) {
+ t.Errorf("board engine cards = %v, want %v", got, want)
+ }
+
+ gui := httptest.NewServer(g.GUIHandler())
+ defer gui.Close()
+ resp := call(t, "GET", gui.URL+"/gui/state", "")
+ defer resp.Body.Close()
+ var rows []guiRow
+ if err := json.NewDecoder(resp.Body).Decode(&rows); err != nil {
+ t.Fatal(err)
+ }
+ got = map[string][]string{}
+ for _, r := range rows {
+ got[r.Name] = r.Cards
+ }
+ if !reflect.DeepEqual(got, want) {
+ t.Errorf("/gui/state cards = %v, want %v", got, want)
+ }
+}
diff --git a/internal/gateway/cardurl_test.go b/internal/gateway/cardurl_test.go
new file mode 100644
index 0000000..853f9a7
--- /dev/null
+++ b/internal/gateway/cardurl_test.go
@@ -0,0 +1,89 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "log/slog"
+ "net/http"
+ "net/http/httptest"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+)
+
+// cardEngine is an engine that serves qwen plus a model of its own, and
+// answers a chat request with its name and the path it was sent to.
+func cardEngine(t *testing.T, name string) *httptest.Server {
+ t.Helper()
+ s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/v1/models":
+ fmt.Fprintf(w, `{"object":"list","data":[{"id":"qwen"},{"id":%q}]}`, name+"-only")
+ default:
+ fmt.Fprintf(w, "%s %s", name, r.URL.Path)
+ }
+ }))
+ t.Cleanup(s.Close)
+ return s
+}
+
+// One model served on two cards: a /cards/{card}/v1 base URL reaches the
+// engine on that card, a card the node lacks is a 404 naming the cards, and
+// that card's model list names only what its engines serve.
+func TestCardBaseURLPicksTheEngineOnThatCard(t *testing.T) {
+ sock, _ := twoCards(t)
+ a, b := cardEngine(t, "on-gpu0"), cardEngine(t, "on-gpu1")
+ g := &Gateway{Socket: sock, Ready: 5 * time.Second, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil)),
+ Engines: []config.Engine{
+ {Name: "qwen-a", Type: "llamacpp", URL: a.URL, GPU: "gpu0", Serves: []string{"qwen", "on-gpu0-only"}},
+ {Name: "qwen-b", Type: "llamacpp", URL: b.URL, GPU: "0000:09:00.0", Serves: []string{"qwen", "on-gpu1-only"}},
+ }}
+ gw := httptest.NewServer(g.Handler())
+ defer gw.Close()
+
+ chat := func(path, model string) (int, string) {
+ resp := call(t, "POST", gw.URL+path, fmt.Sprintf(`{"model":%q}`, model))
+ defer resp.Body.Close()
+ body, _ := io.ReadAll(resp.Body)
+ return resp.StatusCode, string(body)
+ }
+ for _, c := range []struct{ path, model, want string }{
+ {"/cards/gpu1/v1/chat/completions", "qwen", "on-gpu1 /v1/chat/completions"},
+ {"/cards/gpu0/v1/chat/completions", "qwen", "on-gpu0 /v1/chat/completions"},
+ {"/cards/0000:09:00.0/v1/embeddings", "qwen", "on-gpu1 /v1/embeddings"},
+ {"/v1/chat/completions", "qwen", "on-gpu0 /v1/chat/completions"},
+ } {
+ if code, body := chat(c.path, c.model); code != http.StatusOK || body != c.want {
+ t.Errorf("POST %s: %d %q, want 200 %q", c.path, code, body, c.want)
+ }
+ }
+ if code, body := chat("/cards/gpu0/v1/chat/completions", "on-gpu1-only"); code != http.StatusNotFound || !strings.Contains(body, "gpu0") {
+ t.Errorf("a model only gpu1 serves, asked on gpu0: %d %q, want a 404 naming gpu0", code, body)
+ }
+ if code, body := chat("/cards/gpu7/v1/chat/completions", "qwen"); code != http.StatusNotFound || !strings.Contains(body, "cards: gpu0, gpu1") {
+ t.Errorf("an unknown card: %d %q, want a 404 listing the cards", code, body)
+ }
+
+ resp := call(t, "GET", gw.URL+"/cards/gpu1/v1/models", "")
+ defer resp.Body.Close()
+ var list struct{ Data []struct{ ID string } }
+ if err := json.NewDecoder(resp.Body).Decode(&list); err != nil {
+ t.Fatal(err)
+ }
+ var ids []string
+ for _, m := range list.Data {
+ ids = append(ids, m.ID)
+ }
+ if !slices.Equal(ids, []string{"on-gpu1-only", "qwen"}) {
+ t.Errorf("gpu1's models = %v, want [on-gpu1-only qwen]", ids)
+ }
+}
diff --git a/internal/gateway/cordon_test.go b/internal/gateway/cordon_test.go
new file mode 100644
index 0000000..047c6c0
--- /dev/null
+++ b/internal/gateway/cordon_test.go
@@ -0,0 +1,69 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "context"
+ "io"
+ "log/slog"
+ "os"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// A cordoned card is not free: its background engine waits until berth
+// uncordon, and a ComfyUI job for it is told why it cannot run.
+func TestCordonedCardIsNotFree(t *testing.T) {
+ sock, n := twoCards(t)
+ if _, err := n.Cordon("gpu1", true); err != nil {
+ t.Fatal(err)
+ }
+ var st proto.Status
+ until(t, "both cards read", func() bool {
+ st = n.Status()
+ return node.View(&st, "gpu0") != nil && node.View(&st, "gpu1") != nil
+ })
+ if why := outOfReach(nodeView(t, &st, "gpu1")); why != "the card is cordoned" {
+ t.Errorf("outOfReach(gpu1) = %q, want the cordon", why)
+ }
+ if why := outOfReach(nodeView(t, &st, "gpu0")); why != "" {
+ t.Errorf("outOfReach(gpu0) = %q, want nothing", why)
+ }
+
+ addr := freeAddr(t)
+ g := &Gateway{Socket: sock, Ready: 10 * time.Second, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil)),
+ Engines: []config.Engine{{Name: "tagger", Type: "llamacpp", URL: "http://" + addr, GPU: "gpu1",
+ Start: []string{"env", "BERTH_GW_ENGINE=" + addr + ",tagger-model", os.Args[0]},
+ Background: true, IdleAfter: config.Duration{Duration: 50 * time.Millisecond}}}}
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ go g.Background(ctx, 20*time.Millisecond)
+
+ time.Sleep(400 * time.Millisecond)
+ if st := n.Cards()[1].B.Status(); len(st.Leases)+len(st.Waiting) != 0 {
+ t.Fatalf("the gateway started a background run for the cordoned card: leases %+v, waiting %+v", st.Leases, st.Waiting)
+ }
+ if _, err := n.Cordon("gpu1", false); err != nil {
+ t.Fatal(err)
+ }
+ until(t, "the tagger to start once gpu1 is uncordoned", func() bool {
+ ls := n.Cards()[1].B.Status().Leases
+ return len(ls) == 1 && ls[0].Engine == "tagger"
+ })
+}
+
+func nodeView(t *testing.T, st *proto.Status, card string) *proto.Status {
+ t.Helper()
+ v := node.View(st, card)
+ if v == nil {
+ t.Fatalf("no card %s", card)
+ }
+ return v
+}
diff --git a/internal/gateway/deferred.go b/internal/gateway/deferred.go
index 7071c33..fb31bae 100644
--- a/internal/gateway/deferred.go
+++ b/internal/gateway/deferred.go
@@ -19,6 +19,7 @@ import (
"io"
"net/http"
"os"
+ "slices"
"strconv"
"strings"
"time"
@@ -59,6 +60,9 @@ func outOfReach(st *proto.Status) string {
if st.Passthrough != nil {
return "the card is passed through to a VM"
}
+ if cordoned(st) {
+ return "the card is cordoned"
+ }
for _, l := range st.Leases {
if l.Pinned {
return fmt.Sprintf("%s holds the card", holderName(l))
@@ -262,3 +266,8 @@ func (g *Gateway) deferredHistory(w http.ResponseWriter, r *http.Request, id str
}
_ = json.NewEncoder(w).Encode(out)
}
+
+// cordoned is whether berth cordon closed a card the status covers.
+func cordoned(st *proto.Status) bool {
+ return slices.ContainsFunc(st.GPUs, func(g proto.GPU) bool { return g.Scheduled && g.Cordoned })
+}
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index ec96d20..e2e21ac 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -212,7 +212,9 @@ func (g *Gateway) Handler() http.Handler {
g.adoptOnce.Do(g.adopt)
mux := http.NewServeMux()
mux.HandleFunc("GET /v1/models", g.models)
+ mux.HandleFunc("GET /cards/{card}/v1/models", g.onCard(g.models))
mux.HandleFunc("GET /v1/berth/wait", g.wait)
+ mux.HandleFunc("GET /cards/{card}/v1/berth/wait", g.onCard(g.wait))
mux.HandleFunc("GET /v1/berth/cost", g.cost)
mux.HandleFunc("POST /v1/berth/plan", g.plan)
mux.HandleFunc("GET /v1/berth/board", g.board)
@@ -220,6 +222,7 @@ func (g *Gateway) Handler() http.Handler {
mux.HandleFunc("POST /v1/berth/stop", g.apiStop)
for _, p := range []string{"/v1/chat/completions", "/v1/completions", "/v1/embeddings"} {
mux.HandleFunc("POST "+p, g.forward)
+ mux.HandleFunc("POST /cards/{card}"+p, g.onCard(g.forward))
}
mux.HandleFunc("POST /prompt", func(w http.ResponseWriter, r *http.Request) { g.comfyPrompt(w, r, false) })
mux.HandleFunc("/ws", g.comfyProxy)
@@ -384,37 +387,114 @@ func (g *Gateway) listed(ctx context.Context, base string) []string {
return out
}
-// resolve finds the engine for model: an engine that is up and lists it
-// first (it loads the model itself, under its own leases), else a configured
-// engine that serves it and can be started.
-func (g *Gateway) resolve(ctx context.Context, model string) (target, bool) {
+// onCard serves a /cards/{card}/v1/... route: the card is named by its
+// name or PCI address, the path the engine sees loses the prefix, and the
+// handler finds the card's name under PathValue("card").
+func (g *Gateway) onCard(h http.HandlerFunc) http.HandlerFunc {
+ return func(w http.ResponseWriter, r *http.Request) {
+ key := r.PathValue("card")
+ st := g.nodeStatus()
+ if st == nil {
+ apiErrorCode(w, http.StatusServiceUnavailable, "no_berth", "berthd is not answering, so the gateway cannot tell which card "+key+" is")
+ return
+ }
+ cards := node.CardsOf(st, key)
+ if len(cards) != 1 {
+ var names []string
+ for _, c := range st.GPUs {
+ if c.Scheduled {
+ names = append(names, c.Card)
+ }
+ }
+ apiError(w, http.StatusNotFound, fmt.Sprintf("no card %s on this node (cards: %s)", key, strings.Join(names, ", ")))
+ return
+ }
+ r.SetPathValue("card", cards[0])
+ r.URL.Path = r.URL.Path[strings.Index(r.URL.Path, "/v1/"):]
+ r.URL.RawPath = ""
+ h(w, r)
+ }
+}
+
+// engineOn is whether the engine runs on the card named card; "" (no card
+// asked) takes every engine. An engine berthd sees that has no entry in
+// berth.toml is on no card the gateway knows.
+func (g *Gateway) engineOn(st *proto.Status, name, card string) bool {
+ if card == "" {
+ return true
+ }
+ e := g.engineByName(name)
+ return e != nil && st != nil && slices.Contains(node.CardsOf(st, e.GPU), card)
+}
+
+// resolve finds the engine for model, on card when one is named: an engine
+// that is up and lists it first (it loads the model itself, under its own
+// leases), else a configured engine that serves it and can be started.
+func (g *Gateway) resolve(ctx context.Context, model, card string) (target, bool) {
st := g.nodeStatus()
for _, e := range g.live() {
// A parked engine answers its list with the weights in RAM; the
// resume, not the list, says when it may be proxied.
- if e.Up && !parkedIn(st, e.Name) && slices.Contains(g.listed(ctx, e.URL), model) {
+ if e.Up && g.engineOn(st, e.Name, card) && !parkedIn(st, e.Name) && slices.Contains(g.listed(ctx, e.URL), model) {
return target{url: e.URL, live: true}, true
}
}
+ var serving []*config.Engine
for i := range g.Engines {
- e := &g.Engines[i]
- if slices.Contains(e.Serves, model) {
- return target{url: e.URL, engine: e}, true
+ if e := &g.Engines[i]; g.engineOn(st, e.Name, card) && slices.Contains(e.Serves, model) {
+ serving = append(serving, e)
}
}
- return target{}, false
+ if len(serving) == 0 {
+ return target{}, false
+ }
+ // One engine per card serves the model: the cheapest card to reach
+ // wins, and berth.toml's order breaks a tie.
+ e := slices.MinFunc(serving, func(a, b *config.Engine) int { return reach(st, a) - reach(st, b) })
+ return target{url: e.URL, engine: e}, true
+}
+
+// reach ranks what it takes to bring e up on its card, cheapest first: a
+// resume from RAM, a free card, a drain of the holder, then a wait for a
+// card a VM or a pinned lease (a game, a hold) keeps.
+func reach(st *proto.Status, e *config.Engine) int {
+ v := node.View(st, e.GPU)
+ switch {
+ case parkedIn(st, e.Name):
+ return 0
+ case cardFree(v):
+ return 1
+ case v != nil && outOfReach(v) == "":
+ return 2
+ default:
+ return 3
+ }
+}
+
+// notServed is the 404 for a model no engine serves, on card when one is
+// named.
+func notServed(w http.ResponseWriter, model, card string) {
+ if card == "" {
+ apiError(w, http.StatusNotFound, fmt.Sprintf("no engine on this node serves %q (GET /v1/models lists what does)", model))
+ return
+ }
+ apiError(w, http.StatusNotFound, fmt.Sprintf("no engine on %s serves %q (GET /cards/%s/v1/models lists what does)", card, model, card))
}
-func (g *Gateway) catalog(ctx context.Context) []string {
+func (g *Gateway) catalog(ctx context.Context, card string) []string {
+ st := g.nodeStatus()
seen := map[string]bool{}
for _, e := range g.live() {
- if e.Up {
+ if e.Up && g.engineOn(st, e.Name, card) {
for _, m := range g.listed(ctx, e.URL) {
seen[m] = true
}
}
}
for _, e := range g.Engines {
+ if !g.engineOn(st, e.Name, card) {
+ continue
+ }
for _, m := range e.Serves {
seen[m] = true
}
@@ -437,7 +517,7 @@ func (g *Gateway) models(w http.ResponseWriter, r *http.Request) {
Object string `json:"object"`
Data []model `json:"data"`
}{Object: "list", Data: []model{}}
- for _, m := range g.catalog(r.Context()) {
+ for _, m := range g.catalog(r.Context(), r.PathValue("card")) {
out.Data = append(out.Data, model{ID: m, Object: "model", OwnedBy: "berth"})
}
w.Header().Set("Content-Type", "application/json")
@@ -554,9 +634,9 @@ func (g *Gateway) forward(w http.ResponseWriter, r *http.Request) {
apiError(w, http.StatusBadRequest, "the request needs a model")
return
}
- t, ok := g.resolve(r.Context(), req.Model)
+ t, ok := g.resolve(r.Context(), req.Model, r.PathValue("card"))
if !ok {
- apiError(w, http.StatusNotFound, fmt.Sprintf("no engine on this node serves %q (GET /v1/models lists what does)", req.Model))
+ notServed(w, req.Model, r.PathValue("card"))
return
}
preamble := false
@@ -654,9 +734,9 @@ func (g *Gateway) wait(w http.ResponseWriter, r *http.Request) {
apiError(w, http.StatusBadRequest, "name the model to wait for: /v1/berth/wait?model=NAME")
return
}
- t, ok := g.resolve(r.Context(), model)
+ t, ok := g.resolve(r.Context(), model, r.PathValue("card"))
if !ok {
- apiError(w, http.StatusNotFound, fmt.Sprintf("no engine on this node serves %q (GET /v1/models lists what does)", model))
+ notServed(w, model, r.PathValue("card"))
return
}
if !t.live {
diff --git a/internal/gateway/gui.go b/internal/gateway/gui.go
index edf58f9..6332294 100644
--- a/internal/gateway/gui.go
+++ b/internal/gateway/gui.go
@@ -146,6 +146,9 @@ type guiRow struct {
Detail string `json:"detail,omitempty"` // the start's last log line, or its error
Code string `json:"code,omitempty"`
Card []guiHold `json:"card,omitempty"`
+ // Cards are the cards the engine's entry runs on, by name; empty while
+ // berthd does not answer.
+ Cards []string `json:"cards,omitempty"`
}
// guiRowFor is the static part of an engine's row: what it is. The state
@@ -181,6 +184,9 @@ func (g *Gateway) guiRows(ctx context.Context, status *proto.Status) []guiRow {
for i := range g.Engines {
e := &g.Engines[i]
row := guiRowFor(e)
+ if status != nil {
+ row.Cards = node.CardsOf(status, e.GPU)
+ }
if g.probe(ctx, e) {
row.State = "up"
row.Message = e.Name + " is ready"
diff --git a/internal/gateway/gui/index.js b/internal/gateway/gui/index.js
index cc73274..6a8e7cd 100644
--- a/internal/gateway/gui/index.js
+++ b/internal/gateway/gui/index.js
@@ -10,6 +10,18 @@
// The last state each row showed, so a click knows which button it pressed.
const seen = new Map();
+ // On a node with several cards each row names its card, so launching the
+ // copy of a model on gpu1 is how a phone picks the card.
+ function showCards(list) {
+ const several = new Set(list.flatMap((s) => s.cards || [])).size > 1;
+ for (const s of list) {
+ const chip = rows.find((r) => r.dataset.name === s.name)?.querySelector(".gpu");
+ if (!chip) continue;
+ chip.textContent = (s.cards || []).join(", ");
+ chip.hidden = !several || !chip.textContent;
+ }
+ }
+
function showRow(s) {
const li = rows.find((r) => r.dataset.name === s.name);
if (!li) return; // the engine left berth.toml while the page is open
@@ -89,6 +101,7 @@
list = await r.json();
gw.hidden = true;
for (const s of list) showRow(s);
+ showCards(list);
} catch {
gw.hidden = false;
setTimeout(poll, 2000);
diff --git a/internal/gateway/gui/page.html b/internal/gateway/gui/page.html
index 714e55d..85f2356 100644
--- a/internal/gateway/gui/page.html
+++ b/internal/gateway/gui/page.html
@@ -27,6 +27,7 @@
berth
{{- if and .GUI .Launch}}{{.Name}}{{else}}{{.Name}}{{end}}
{{- if not .GUI}}No UI{{end}}
+
diff --git a/internal/gateway/gui/wake.css b/internal/gateway/gui/wake.css
index 46986a5..9b34faa 100644
--- a/internal/gateway/gui/wake.css
+++ b/internal/gateway/gui/wake.css
@@ -78,6 +78,7 @@ button:focus-visible, a:focus-visible { outline: 2px solid var(--cyan); outline-
.row-main a { color: var(--cyan); text-decoration: none; }
.row-main a:hover { text-decoration: underline; }
.chip.noui { background: transparent; color: var(--dim); border: 1px solid var(--hair); }
+.chip.gpu { background: transparent; color: var(--cyan); border: 1px solid var(--cyan); }
.chip.up { background: var(--green); }
.chip.offline { background: var(--dim); }
.chip.waiting, .chip.starting { background: var(--cyan); animation: pulse 1.2s ease-in-out infinite; }
diff --git a/internal/gateway/placement_test.go b/internal/gateway/placement_test.go
new file mode 100644
index 0000000..0da85ab
--- /dev/null
+++ b/internal/gateway/placement_test.go
@@ -0,0 +1,89 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "log/slog"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/procs"
+)
+
+// A plain /v1 request for a model served on two cards goes to the card that
+// costs least to reach: a parked copy before a free card, a free card before
+// one a lease holds, and a held card before one a game has pinned. With
+// nothing to tell them apart, the first entry in berth.toml wins.
+func TestPlainRequestPicksTheCheapestCard(t *testing.T) {
+ for _, c := range []struct {
+ name string
+ hold func(t *testing.T, gpu0, gpu1 *broker.Broker)
+ want string
+ }{
+ {"both free", nil, "on-gpu0"},
+ {"a game pins gpu0", func(t *testing.T, gpu0, _ *broker.Broker) {
+ acquire(t, gpu0, broker.Request{Engine: "game", Priority: broker.Hold, Pinned: true})
+ }, "on-gpu1"},
+ {"a lease holds gpu0", func(t *testing.T, gpu0, _ *broker.Broker) {
+ acquire(t, gpu0, broker.Request{Engine: "tts", Priority: broker.Batch})
+ }, "on-gpu1"},
+ {"a game pins gpu0, a lease holds gpu1", func(t *testing.T, gpu0, gpu1 *broker.Broker) {
+ acquire(t, gpu0, broker.Request{Engine: "game", Priority: broker.Hold, Pinned: true})
+ acquire(t, gpu1, broker.Request{Engine: "tts", Priority: broker.Batch})
+ }, "on-gpu1"},
+ {"gpu0 cordoned", func(t *testing.T, gpu0, _ *broker.Broker) {
+ gpu0.Cordon(true)
+ }, "on-gpu1"},
+ {"qwen parked on gpu1", func(t *testing.T, _, gpu1 *broker.Broker) {
+ pid := os.Getpid()
+ l, err := gpu1.Reattach(broker.Request{Engine: "qwen-b", Parked: true}, pid, procs.System{}.StartTime(pid))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(l.Release)
+ }, "on-gpu1"},
+ } {
+ t.Run(c.name, func(t *testing.T) {
+ sock, n := twoCards(t)
+ if c.hold != nil {
+ c.hold(t, n.Cards()[0].B, n.Cards()[1].B)
+ }
+ a, b := cardEngine(t, "on-gpu0"), cardEngine(t, "on-gpu1")
+ g := &Gateway{Socket: sock, Ready: 5 * time.Second, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil)),
+ Engines: []config.Engine{
+ {Name: "qwen-a", Type: "llamacpp", URL: a.URL, GPU: "gpu0", Serves: []string{"qwen"}},
+ {Name: "qwen-b", Type: "llamacpp", URL: b.URL, GPU: "gpu1", Serves: []string{"qwen"}},
+ }}
+ gw := httptest.NewServer(g.Handler())
+ defer gw.Close()
+ resp := call(t, "POST", gw.URL+"/v1/chat/completions", `{"model":"qwen"}`)
+ defer resp.Body.Close()
+ body, _ := io.ReadAll(resp.Body)
+ if want := c.want + " /v1/chat/completions"; resp.StatusCode != http.StatusOK || string(body) != want {
+ t.Errorf("POST /v1/chat/completions: %d %q, want 200 %q", resp.StatusCode, body, want)
+ }
+ })
+ }
+}
+
+// acquire takes a lease on b for the rest of the test.
+func acquire(t *testing.T, b *broker.Broker, r broker.Request) {
+ t.Helper()
+ r.Wait = 5 * time.Second
+ l, err := b.Acquire(context.Background(), r, nil)
+ if err != nil {
+ t.Fatal(fmt.Errorf("acquire %s: %w", r.Engine, err))
+ }
+ t.Cleanup(l.Release)
+}
diff --git a/internal/gateway/session_test.go b/internal/gateway/session_test.go
index a77ec1b..3a829d7 100644
--- a/internal/gateway/session_test.go
+++ b/internal/gateway/session_test.go
@@ -36,3 +36,19 @@ func TestRunArgvCarriesTheCard(t *testing.T) {
t.Fatalf("an entry with no card gained one: %q", bare)
}
}
+
+// A detached gateway launch carries the entry's slice: the engine scope
+// the run creates takes it, so the slice's policy covers the engine.
+func TestRunArgvCarriesSlice(t *testing.T) {
+ e := config.Engine{Name: "fixture", Slice: "strata.slice"}
+ o := runner.OptionsFor(&e, runner.Over{})
+ args := runArgv(o)
+ i := slices.Index(args, "--slice")
+ if i < 0 || i+1 >= len(args) || args[i+1] != "strata.slice" {
+ t.Fatalf("lost slice: %q", args)
+ }
+ bare := runArgv(runner.OptionsFor(&config.Engine{Name: "plain"}, runner.Over{}))
+ if slices.Contains(bare, "--slice") {
+ t.Fatalf("an entry with no slice gained one: %q", bare)
+ }
+}
diff --git a/internal/gateway/spawn.go b/internal/gateway/spawn.go
index c153bb7..ad86307 100644
--- a/internal/gateway/spawn.go
+++ b/internal/gateway/spawn.go
@@ -47,7 +47,9 @@ func (g *Gateway) spawnRun(name string, s *started, o runner.Options) (int, erro
}
full = append(full, "--")
full = append(full, o.Argv...)
- full = sysd.ScopeArgv("berth-engine", name, full)
+ // The gateway's own run wrapper: the engine's scope comes from the
+ // run's --slice, which runArgv carries from the entry.
+ full = sysd.ScopeArgv("berth-engine", name, "", full)
cmd := exec.Command(full[0], full[1:]...)
detach(cmd)
cmd.Stdout, cmd.Stderr = logFile, logFile
@@ -130,6 +132,9 @@ func runArgv(o runner.Options) []string {
if o.FailClosed {
args = append(args, "--fail-closed")
}
+ if o.Slice != "" {
+ args = append(args, "--slice", o.Slice)
+ }
return args
}
diff --git a/internal/harbor/web/demo.json b/internal/harbor/web/demo.json
index 17a0695..0e6951e 100644
--- a/internal/harbor/web/demo.json
+++ b/internal/harbor/web/demo.json
@@ -151,6 +151,46 @@
"estimate_s": 120
}
],
+ "claims": [
+ {
+ "name": "steam-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "lutris-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "gamescope",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "blender",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "obs",
+ "action": "track",
+ "builtin": true,
+ "active": true,
+ "pids": [
+ 48211
+ ],
+ "procs": "obs",
+ "used": 644245094,
+ "cards": [
+ "gpu0"
+ ]
+ }
+ ],
"squatters": [
{
"pid": 60233,
diff --git a/internal/harbor/web/harbor.css b/internal/harbor/web/harbor.css
index 96ea9f1..8c7711f 100644
--- a/internal/harbor/web/harbor.css
+++ b/internal/harbor/web/harbor.css
@@ -310,7 +310,10 @@ body.mode-basic .adv-only { display: none !important; }
.mode-btn + .mode-btn { border-left: 1px solid var(--hair-2); }
.mode-btn[aria-pressed="true"] { color: var(--cyan); background: var(--cyan-wash); }
.mode-btn:hover { color: var(--ink); }
-.tabs { display: flex; gap: 4px; margin-left: auto; }
+.tabs { display: flex; gap: 4px; margin-left: auto; flex: none; }
+/* the mode switch and the tab words never shrink: the descriptor gives way first */
+.app-header .mode { flex: none; }
+.tab-label { white-space: nowrap; }
.tab {
appearance: none;
background: transparent;
@@ -339,7 +342,7 @@ body.mode-basic .adv-only { display: none !important; }
padding: 0 6px; line-height: 1.6;
}
.tab[aria-current="page"] .count { color: var(--amber); border-color: var(--amber-dim); }
-.header-status { display: flex; align-items: center; gap: var(--s3); flex-wrap: wrap; }
+.header-status { display: flex; align-items: center; justify-content: flex-end; gap: var(--s2) var(--s3); flex-wrap: wrap; }
.node-version { font-family: var(--mono); font-size: var(--fs-small); color: var(--dim); }
.update-notice { display: inline-flex; align-items: center; gap: 6px; font-family: var(--mono); font-size: var(--fs-small); }
.update-notice .update-link { color: var(--cyan); text-decoration: none; }
@@ -1020,6 +1023,13 @@ body.ready #panel-engines > :nth-child(n+2) { animation-delay: 60ms; }
.sq .name { font-size: var(--fs-body); color: var(--magenta); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.sq .meta, .nr .meta { color: var(--dim); font-variant-numeric: tabular-nums; overflow-wrap: anywhere; }
.sq .noticket { color: var(--amber); }
+/* a claim is not a stowaway: a hold reads in the attention amber, a track in
+ the passive cyan, and the rule's name never in the stowaway magenta */
+.sq.claim-hold { border-left-color: var(--amber); }
+.sq.claim-track { border-left-color: var(--cyan); }
+.sq.claim-hold .name { color: var(--amber); }
+.sq.claim-track .name { color: var(--cyan); }
+.sq .claim-builtin { color: var(--dim-2); font-size: var(--fs-small); }
.empty-note {
border: 1px dashed var(--hair-2);
border-radius: var(--r1);
@@ -1606,16 +1616,48 @@ dialog.sheet::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
.sheet-hint { margin-left: auto; color: var(--dim-2); font-size: var(--fs-micro); letter-spacing: 0; }
/* ---------- about ---------- */
-.about {
- border: 1px solid var(--hair);
+/* a lead card, then the reference sections as cards that flow down two
+ columns on a wide screen; the rail above jumps between them */
+.about-hero {
+ border: 1px solid var(--hair-2);
border-radius: var(--r2);
- background: linear-gradient(180deg, var(--panel-hi) 0, var(--panel) 132px);
- box-shadow: var(--shadow), inset 0 1px 0 rgba(255, 255, 255, 0.035);
+ background: linear-gradient(180deg, var(--panel-hi) 0, var(--panel) 160px);
+ box-shadow: var(--e2), inset 0 1px 0 rgba(255, 255, 255, 0.035);
padding: var(--s6);
display: flex;
flex-direction: column;
gap: var(--s4);
- max-width: 920px;
+ margin-bottom: var(--s5);
+}
+.about-hero p { margin: 0; color: var(--ink); font-size: var(--fs-body); line-height: 1.6; max-width: 88ch; }
+.about-hero .about-word { display: flex; flex-wrap: wrap; align-items: baseline; gap: var(--s2) var(--s3); color: var(--dim); font-size: var(--fs-label); }
+.about-facts { display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: 4px var(--s4); margin: 0; font-family: var(--mono); font-size: var(--fs-small); }
+.about-facts dt { color: var(--dim-2); letter-spacing: 1px; text-transform: uppercase; }
+.about-facts dd { margin: 0; color: var(--ink); overflow-wrap: anywhere; }
+.about-links { display: flex; flex-wrap: wrap; gap: var(--s2); margin: 0; padding: 0; list-style: none; }
+.about-links a, .about-sec a {
+ color: var(--cyan); text-decoration: underline; text-decoration-color: var(--cyan-dim);
+ text-underline-offset: 3px;
+}
+.about-links a {
+ display: inline-flex; align-items: center; min-height: 36px; padding: 6px var(--s3);
+ border: 1px solid var(--hair-2); border-radius: var(--r1); background: var(--panel-2);
+ font-family: var(--mono); font-size: var(--fs-small); letter-spacing: 1px; text-decoration: none;
+}
+.about-links a:hover, .about-sec a:hover { color: var(--ink-strong); border-color: var(--cyan-dim); }
+.about-links a:focus-visible, .about-sec a:focus-visible { outline: 2px solid var(--cyan); outline-offset: 2px; }
+.about { columns: 2 520px; column-gap: var(--s5); }
+.about-sec {
+ break-inside: avoid;
+ border: 1px solid var(--hair);
+ border-radius: var(--r2);
+ background: var(--panel);
+ box-shadow: var(--e1);
+ padding: var(--s5);
+ margin: 0 0 var(--s5);
+ display: flex;
+ flex-direction: column;
+ gap: var(--s3);
}
.about .section-title {
font-family: var(--mono);
@@ -1623,14 +1665,17 @@ dialog.sheet::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
letter-spacing: 2px;
color: var(--amber);
font-weight: 600;
- margin: var(--s3) 0 0 0;
-}
-.about p { margin: 0; color: var(--ink); font-size: var(--fs-body); max-width: 80ch; }
-.about-dl { display: grid; grid-template-columns: 170px minmax(0, 1fr); gap: 6px var(--s4); margin: 0; }
-.about-dl dt { font-family: var(--mono); font-size: var(--fs-small); letter-spacing: 1px; color: var(--amber); padding-top: 2px; }
-.about-dl dd { margin: 0; font-size: var(--fs-body); color: var(--dim); }
-.about-ul { margin: 0; padding-left: 18px; color: var(--dim); font-size: var(--fs-body); }
-.about-ul li { margin: 4px 0; }
+ margin: 0;
+}
+.about p { margin: 0; color: var(--ink); font-size: var(--fs-body); line-height: 1.6; }
+.about-dl { display: grid; grid-template-columns: minmax(96px, max-content) minmax(0, 1fr); gap: 8px var(--s4); margin: 0; }
+.about-dl dt { font-family: var(--mono); font-size: var(--fs-small); letter-spacing: 1px; color: var(--amber); padding-top: 2px; white-space: nowrap; }
+.about-dl dd { margin: 0; font-size: var(--fs-body); color: var(--dim); line-height: 1.5; }
+.about-keys kbd {
+ display: inline-block; min-width: 22px; padding: 1px 6px; text-align: center;
+ font-family: var(--mono); font-size: var(--fs-small); color: var(--ink);
+ border: 1px solid var(--hair-2); border-bottom-width: 2px; border-radius: var(--r1); background: var(--panel-3);
+}
.about-pre {
margin: 0;
font-family: var(--mono);
@@ -1658,6 +1703,11 @@ dialog.sheet::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
}
/* ---------- responsive: one query per breakpoint ---------- */
+@media (max-width: 1599px) {
+ /* below this the descriptor only fits cut off; the wordmark names the app
+ and the about tab leads with the same line */
+ .tab-long, .descriptor { display: none; }
+}
@media (max-width: 1279px) {
.harbor-grid { grid-template-columns: minmax(0, 7fr) minmax(300px, 5fr); }
}
@@ -1805,6 +1855,11 @@ dialog.sheet::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
should not have to aim at a 24 px glyph */
.toast-close, .conn .retry, .alert-card .alert-dismiss { min-width: var(--touch); min-height: var(--touch); }
.alert-card .alert-dismiss { margin: -8px -6px -8px 0; }
+ /* an alert's message is the warning itself: on a phone it wraps instead of
+ ending at "2 waitin..." */
+ .alert-card { flex-wrap: wrap; row-gap: 2px; }
+ .alert-card .alert-msg { white-space: normal; overflow-wrap: anywhere; flex: 1 1 0; }
+ .alert-card .alert-when { order: 9; flex-basis: 100%; }
.conn .retry { margin: -6px -8px -6px 0; }
/* every chart's shape, not its words, takes the phone's minimum height: an
axis label has had room to be 13 px since the frame stopped scaling text */
@@ -1846,6 +1901,7 @@ dialog.sheet::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
.ticket-stub { display: none; }
.ticket-paper { --fold-x: 100%; }
.about-dl { grid-template-columns: 1fr; }
+ .about-hero, .about-sec { padding: var(--s4); }
}
/* ---------- motion preferences and other media ---------- */
diff --git a/internal/harbor/web/harbor.js b/internal/harbor/web/harbor.js
index 30d7b69..6bf5e0d 100644
--- a/internal/harbor/web/harbor.js
+++ b/internal/harbor/web/harbor.js
@@ -132,7 +132,7 @@ let shortcutsDlg, railMenuDlg;
let lastLogSig = null; // filter+query+pause signature: repaint only on change, append otherwise
let clearLogFilter = null; // set by buildLogSearch: the page's Esc reaches it (R-07)
let lastTick = 0; // the relative-time ticker's own 1 s cadence
-let berthsRoot, stowRoot, notickRoot, machineRoot, machineCharts, machineMeta, socketLine;
+let berthsRoot, claimsRoot, claimsMeta, stowRoot, notickRoot, machineRoot, machineCharts, machineMeta, socketLine;
let kpiStripEl, tideEl, tideLegend, tideMeta;
let boardMeta, berthsMeta, stowMeta, noticketMeta, enginesMeta;
let throughputMeta, throughputDecode, throughputRpm;
@@ -167,6 +167,8 @@ function main() {
shortcutsDlg = $("shortcuts");
railMenuDlg = $("rail-menu");
berthsRoot = $("berths");
+ claimsRoot = $("claims");
+ claimsMeta = $("claims-meta");
stowRoot = $("stowaways");
notickRoot = $("noticket");
machineRoot = $("machine");
@@ -483,10 +485,19 @@ function wireTabs() {
show(key);
});
}
+ // A hash is a tab, a ticket, or a section the rail wrote: a section opens
+ // the tab that holds it, so a shared or reloaded link lands where it says.
const h = location.hash.slice(1);
- show(h.startsWith("ticket=") ? "harbor" : h);
+ if (tabs[h] || h.startsWith("ticket=")) return show(h);
+ const target = h ? document.getElementById(h) : null;
+ const key = target ? Object.keys(panels).find((k) => panels[k].contains(target)) : undefined;
+ show(key);
+ if (key) pendingSection = target;
}
let pendingTicket = location.hash.startsWith("#ticket=") ? decodeURIComponent(location.hash.slice(8)) : null;
+// A linked section waits for the first status, whose alerts and panels move
+// it down the page.
+let pendingSection = null;
// ---------------- page chrome ----------------
// The sub-navigation, the collapsible panels, the keyboard shortcuts, the
@@ -495,13 +506,19 @@ let pendingTicket = location.hash.startsWith("#ticket=") ? decodeURIComponent(lo
const HARBOR_SECTIONS = [
["sec-counter", "Counter"], ["sec-tide", "Tide"], ["sec-timeline", "Timeline"],
["sec-board", "Board"], ["sec-log", "Log"], ["sec-berths", "Berths"],
- ["sec-machine", "Machine"], ["sec-stow", "Stowaways"], ["sec-noticket", "No ticket"],
+ ["sec-machine", "Machine"], ["sec-claims", "Claims"], ["sec-stow", "Stowaways"], ["sec-noticket", "No ticket"],
];
const ENGINES_SECTIONS = [
["sec-engines", "Engines"], ["sec-library", "Library"], ["priority-panel", "Ranking"], ["sec-pool", "KV pool"],
["sec-throughput", "Throughput"], ["sec-ledger", "Ledger"], ["sec-lb", "Leaderboard"],
["sec-recent", "Recent"], ["sec-residency", "Residency"], ["sec-storage", "Storage"],
];
+const ABOUT_SECTIONS = [
+ ["about-top", "Berth"], ["about-cards", "Cards"], ["about-modes", "Modes"], ["about-nav", "Getting around"],
+ ["about-keys", "Keyboard"], ["about-states", "Status words"], ["about-kpis", "KPIs"], ["about-counters", "Counters"],
+ ["about-kv", "Context and KV"], ["about-telemetry", "GPU and machine"], ["about-library", "Library"],
+ ["about-holders", "Claims and stowaways"], ["about-next", "Terminal"],
+];
function wireChrome() {
try {
const stored = localStorage.getItem(COLLAPSE_KEY);
@@ -509,6 +526,7 @@ function wireChrome() {
} catch (e) { state.collapsed = []; }
buildSubnav($("subnav-harbor"), HARBOR_SECTIONS);
buildSubnav($("subnav-engines"), ENGINES_SECTIONS);
+ buildSubnav($("subnav-about"), ABOUT_SECTIONS);
wireCollapse();
wireShortcuts();
wireRailMenu();
@@ -676,6 +694,7 @@ function paintSkeletons() {
skel(logList, 6);
skel(berthsRoot, 3);
skel(machineRoot, 3);
+ skel(claimsRoot, 1);
skel(stowRoot, 2);
skel(notickRoot, 2);
skel($("engine-grid"), 2);
@@ -1185,8 +1204,14 @@ function demoInsights(range) {
}
function onStatus(s) {
nodeVersion.textContent = "node " + (s.version || "") + (s.harbor_version ? " / harbor " + s.harbor_version : "");
+ setText($("about-version"), nodeVersion.textContent);
renderAll(s);
paintOverall();
+ if (pendingSection) {
+ const target = pendingSection;
+ pendingSection = null;
+ requestAnimationFrame(() => target.scrollIntoView({ block: "start" }));
+ }
if (pendingTicket) {
const key = pendingTicket;
pendingTicket = null;
@@ -2057,6 +2082,39 @@ function chartCard(label, ctl) {
return el("div", "machine-chart", el("div", "mc-label", label), ctl.el);
}
+// ---------------- claims ----------------
+// The claim rules matching right now, as berth claims lists them: a hold
+// keeps its cards for a game, a track names an app's card memory. Idle
+// rules are only counted.
+function claimLine(cl) {
+ if (cl.action === "track") return (cl.used || cl.gtt_used) ? memoryWord(cl) + " named, not a stowaway" : "named; holds no card memory now";
+ return cl.lease ? "holds the card as " + cl.lease + "; engines wait" : "waiting for the card";
+}
+function memoryWord(cl) {
+ return fmtGiB(cl.used || 0) + " GiB" + (cl.gtt_used ? " + " + fmtGiB(cl.gtt_used) + " GiB GTT" : "");
+}
+function renderClaims(s) {
+ const list = el("div", "squatter-list");
+ const all = s.claims || [];
+ const active = all.filter((cl) => cl.active);
+ claimsMeta.textContent = active.length ? active.length + " active of " + all.length + " rules" : all.length + " rules, none active";
+ if (all.length === 0) list.append(el("div", "empty-note", "no claim rules: the built-in pack is off and berth.toml has no [[claim]]"));
+ else if (active.length === 0) list.append(el("div", "empty-note", "no game or app is claiming a card"));
+ for (const cl of active) {
+ const on = cardTag(s, (cl.cards || []).join(", "));
+ const name = el("span", "name", cl.name);
+ if (cl.builtin) name.append(el("span", "claim-builtin", " built-in"));
+ const row = el("div", "sq claim " + (cl.action === "track" ? "claim-track" : "claim-hold"));
+ row.append(
+ el("div", "top", name, el("span", null, (on ? on + ", " : "") + (cl.action || "hold").toUpperCase())),
+ el("div", "meta", claimLine(cl)),
+ );
+ if (cl.procs) row.append(el("div", "meta claim-procs", cl.procs));
+ list.append(row);
+ }
+ claimsRoot.replaceChildren(list);
+}
+
// ---------------- stowaways and no ticket ----------------
function renderStowaways(s) {
const list = el("div", "squatter-list");
@@ -2407,6 +2465,7 @@ function renderLight(s) {
paintTabCounts(entries.length, (s.engines || []).length);
renderBerths(s);
paintAlerts(s);
+ renderClaims(s);
renderStowaways(s);
renderNoTicket(s);
renderMachine(s);
diff --git a/internal/harbor/web/index.html b/internal/harbor/web/index.html
index ceae2f3..dd0a2b4 100644
--- a/internal/harbor/web/index.html
+++ b/internal/harbor/web/index.html
@@ -167,6 +167,13 @@
THE MACHINE
+
+
+
CLAIMS
+ games and apps
+
+
+
STOWAWAYS
@@ -281,100 +288,132 @@
STORAGE
-
-
WHAT THIS PAGE IS
-
Berth is the VRAM leasing service for every GPU you run. Engines ask the berthd node for a lease on the card; the node queues the request, asks the current holder to drain, waits until the driver shows the memory free, and only then grants the berth. Nothing is ever granted on process exit alone, and nothing is claimed as reserved unless the driver measured it. Every stat on this page is collected by the node itself, from the driver, the host, and the engines' own endpoints.
-
-
THE TWO MODES
-
-
BASIC
the at-a-glance story: the counters, the request KPIs, the tide, the departures board, and the berths. The mode you land on.
-
ADVANCED
everything a power user wants: full GPU telemetry (clocks, fan, PCIe), host CPU and memory trends, prefill and reingest KPIs, the token ledger, the leaderboard, and the full recent-request table. Toggled in the header, remembered between visits.
-
-
-
GETTING AROUND
-
A section rail under the header jumps to any panel on the tab and marks the one in view. Every panel folds from the chevron in its head and stays folded between visits. The card tab counts voyages and the engines tab counts engines, so the header says which tab wants attention. On a phone the nav sits along the bottom edge with the tab you are on marked, and the menu button opens the node's versions, the last successful read, the detail level, the retry and this list. Times read relative, with the exact clock in the title. When a read goes stale a retry appears on the conn pill that reads the node now instead of waiting out the poll.
-
The log's filter box matches the kind word, the engine and the detail; the chips carry live counts and the count line names what is shown. LATEST jumps back to the newest row. The board's headers sort by vessel, hold, status and age, and the foot names the order. Alert cards dismiss until they clear or change level. The ticket copies its JSON or a deep link to this exact voyage.
-
-
KEYBOARD
-
-
1 / 2 / 3
card, engines, about
-
B / A
basic, advanced mode
-
P
pause or resume the log
-
/
filter the log
-
R
read the node now
-
?
the shortcut list
-
ESC
close a dialog, else clear the log filter
+
+
+
berth. the VRAM leasing service for every GPU you run
+
Engines ask the berthd node for a lease on a card. The node queues the request, asks the current holder to drain, waits until the driver shows the memory free, and only then grants the berth. Nothing is granted on a process exit alone, and nothing is called reserved unless the driver measured it. Every number on this page comes from the node: the driver, the host and the engines' own endpoints.
+
+
this node
not read yet
+
this page
read only: it shows the node and changes nothing
-
-
THE STATUS WORDS
-
-
BOARDING
the request is waiting in the queue for a free berth
-
DOCKED
the lease is active and the engine is using the card, running from its HOT weights in VRAM
-
CASTING OFF
the holder has been asked to drain and is finishing up
-
PARKING
a draining engine is moving its weights to RAM instead of dropping them; a transitional state, resolved once the driver shows the residual gone
-
WARM
the engine is parked: its weights hold in RAM, only a small residual keeps it on the card, and a resume puts it back at its own priority with no reload
-
ADRIFT
the lease is being released and the memory is on its way out
-
AGROUND
the release stalled; the memory is not coming back on its own
-
-
Every state shows as a shape and its word together, the same on every panel: a dot is working now, a ring is waiting or passive, a half-filled circle is in transition, a triangle needs attention, a square is not running or not live. A magenta CONTESTED flag on a docked row means the driver saw other VRAM holders on the card at grant time, so the measured hold and the free space disagree. A green WARM flag marks a parked engine, and its hover text carries the estimated RAM it holds and how long it has been parked; a boarding row carries a Q flag with its place in the queue, and the node's completion estimate when it sent one. The row title, the berth panel and the ticket stamp carry how much each holds, and the ticket names the tier the weights live in, HOT on the card or WARM in RAM, beside the exact terminal order for it as a copy button. CARD RANKING on the engines tab shows how the node orders engines when two at one priority ask for the card inside the juggle window: the config order, or card hours over the last seven days.
-
-
THE REQUEST KPIs
-
-
DECODE t/s
tokens per second generated across the running engines; the sparkline sums every engine's decode rate over the last hour
-
PREFILL t/s
prompt tokens processed per second, the engine's intake rate while it reads a request in
-
TPOT ms
time per output token, the inverse of decode rate, over the last fifteen minutes
-
P95 ms
the request duration that ninety-five percent of recent requests finished within, last fifteen minutes
-
P50 ms
the median request duration, last fifteen minutes
-
REQ/MIN
requests completed per minute over the last five minutes
-
CACHE %
share of prompt tokens served from the KV cache instead of re-run, last hour
-
REINGEST %
the complement of cache, the share of prompt tokens that had to be computed from scratch
-
DRAFT ACCEPT %
of the tokens a speculative draft model proposed, the share the main model accepted, so no work was wasted
-
PREFILL ms
derived time-to-first-token: uncached prompt tokens divided by the prefill rate, used where the engine does not report it directly
-
QUEUE
engines waiting in the berth's own queue for a free berth on the card; each engine's own queue - the requests it holds back - is on its card
-
-
-
THE COUNTERS
-
-
COLLISIONS AVOIDED
grants that waited for a held card instead of loading on top of the holder
-
VRAM HANDED OFF
peak measured bytes of every lease whose memory the card then showed as freed
-
DRAINS
holders asked to give the berth back
-
AGROUND CAUGHT
stuck releases caught before the card stayed dirty
-
STOWAWAYS CAUGHT
VRAM holders found on the card with no lease
-
CONTESTED
grants made while another holder shared the card
-
REFUSED
requests turned away after their wait expired
-
-
Every count is an event the node actually saw. Berth never estimates what might have happened.
-
-
CONTEXT AND KV
-
Context residency draws the engine's own per-request token figures - cached, fresh prefill and generated - as columns against the model's context window, so a request that would not fit its window is visible at a glance; below the columns sit the engines' live KV sessions. The KV pool is each engine's live KV slots (the llama.cpp family's /slots): how many context tokens each session holds against its window, with 75 and 90 percent marks. Stored ranges show the per-minute peak slot fill the harbor folded from the node's samples; the live slots stay live, whatever the range says. An engine without these endpoints reports no window and no slots, and the panels say so rather than guess.
-
-
GPU TELEMETRY AND THE MACHINE
-
The scheduled card reports its power draw against its power limit, temperature, utilisation, and in advanced mode its SM and memory clocks, fan, and PCIe generation and width. A reading of n/a means the driver did not report it, not zero. The machine panel is the host itself: CPU, cores, used and total memory, swap, and the one, five and fifteen minute load averages. Its socket line counts event-stream write timeouts and dropped slow readers, so a stalled log on a live node starts there.
-
-
THE LIBRARY
-
The library lists the model file roots this machine has, the same rows berth models prints: each model's shelf, quant, size, and its CACHED share with the tier it reads as, COOL at half the model's bytes in the kernel page cache or better, COLD below. The kernel takes those pages back whenever the machine needs the RAM, so the share says a reload may skip the disk; it is an estimate of reload speed, never a statement about RAM held. SERVED BY names each engine that reports the model resident now, and EST VRAM is that engine's learned footprint from its past runs, an estimate of what a run needs and never a reservation. Each row's copy buttons give the exact commands: berth prefetch NAME warms the files, berth models show NAME reads the full header.
-
-
STOWAWAYS AND NO TICKET
-
A stowaway is any process holding VRAM without a lease. Berth reports it and never kills it. A "no ticket" run is an engine started while the node was down, so it holds the card outside any lease. Both show here read only.
-
-
THIS RELEASE IS READ ONLY
-
The harbor shows the node. Ordering work is coming in later phases, and these rows are marked coming, not working:
-
-
DRAIN (coming): order the holder to give the berth back from here
-
BERTH HOLD (coming): pin the card for a known window
-
OFFERS (coming): barter a waiting claim with a holder, consent first, never taken without acceptance
-
The LIBRARY listing is on the engines tab: every model file on the machine, its shelf, quant and size, how much of it the kernel page cache holds, and the engine serving it. Still coming: verify state, update offers and the migration plan. Nothing applies on its own.
-
ENGINE MANAGEMENT (coming): each engine's installed version, with install, update and rollback beside it
-
GATEWAY (coming): the one local API's routes, the requests it served and refused, and the client keys that made them
-
SCENES (coming): the node's scene definitions, and the switch that moves the card from one to another
-
JOURNAL (coming): the entries behind changes, each with its dry-run flag and its exact reverse
-
GUI PLANE (coming): engines with a UI start from that app on its own address; this page links to it and never proxies it
Until then, everything the page shows is also on the terminal:
-
berth status
+
+
+
+
+
SEVERAL CARDS
+
One berthd governs every card in the machine. Each card it schedules has its own berth panel with only that card's leases and stowaways, and a card it only reads is marked INVENTORY ONLY. A board row, a stowaway and a lease line name their card (gpu1), and a lease on several cards lists them. The tide draws the memory of all scheduled cards together. A card passed to a VM is marked on its own panel, and the others keep serving.
+
+
+
+
BASIC AND ADVANCED
+
+
BASIC
the counters, the request KPIs, the tide, the departures board and the berths. The page opens here.
+
ADVANCED
adds full GPU telemetry (clocks, fan, PCIe), host CPU and memory trends, prefill and reingest KPIs, the token ledger, the leaderboard and the full request table. The header switch sets it, and the page remembers it.
+
+
+
+
+
GETTING AROUND
+
The rail under the header jumps to any section of the tab and marks the one in view. A section link opens the right tab when you share or reload it. Every panel folds from the chevron in its head and stays folded between visits. The tab counts say how many voyages and engines each tab holds.
+
On a phone the tabs sit along the bottom edge, and the menu button opens the node's versions, the last successful read, the detail level and the retry. Times read relative, with the exact clock in the title. When a read goes stale, a retry on the connection pill reads the node at once.
+
The log's filter matches the kind, the engine and the detail, and the chips carry live counts. LATEST jumps back to the newest row. The board sorts by vessel, hold, status and age. An alert card stays dismissed until it clears or changes level. A ticket copies its JSON or a link to that voyage.
+
+
+
+
KEYBOARD
+
+
123
card, engines, about
+
BA
basic, advanced
+
P
pause or resume the log
+
/
filter the log
+
R
read the node now
+
?
the shortcut list
+
Esc
close a dialog, else clear the log filter
+
+
+
+
+
THE STATUS WORDS
+
+
BOARDING
waiting in the queue for a free berth
+
DOCKED
the lease is active and the engine runs from its HOT weights in VRAM
+
CASTING OFF
the holder was asked to drain and is finishing
+
PARKING
a draining engine is moving its weights to RAM instead of dropping them, until the driver shows the residual gone
+
WARM
parked: the weights are in RAM, a small residual stays on the card, and a resume brings it back at its own priority with no reload
+
ADRIFT
the lease is released and the memory is on its way out
+
AGROUND
the release stalled, and the memory is not coming back on its own
+
+
Each state shows as a shape and its word together on every panel. A dot is working, a ring is waiting, a half-filled circle is in transition, a triangle needs attention, and a square is stopped. A magenta CONTESTED flag on a docked row means the driver saw other VRAM holders at grant time. A green WARM flag marks a parked engine, and its title gives the RAM it holds and how long it has been parked. A boarding row carries a Q flag with its place in the queue and the node's estimate when it sent one. A ticket names the tier the weights live in, HOT or WARM, and gives the exact terminal command as a copy button. CARD RANKING on the engines tab shows how the node orders two engines at one priority inside the juggle window: the config order, or card hours over seven days.
+
+
+
+
THE REQUEST KPIs
+
+
DECODE t/s
tokens generated per second across the running engines; the sparkline is the last hour
+
PREFILL t/s
prompt tokens read per second
+
TPOT ms
time per output token, last fifteen minutes
+
P95 ms
the duration 95% of recent requests finished within, last fifteen minutes
+
P50 ms
the median request duration, last fifteen minutes
+
REQ/MIN
requests completed per minute, last five minutes
+
CACHE %
prompt tokens served from the KV cache, last hour
+
REINGEST %
prompt tokens computed from scratch, the rest of the cache share
+
DRAFT ACCEPT %
draft-model tokens the main model accepted
+
PREFILL ms
time to first token, from uncached prompt tokens and the prefill rate, where the engine does not report it
+
QUEUE
engines waiting in the node's queue for a berth. Each engine's own request queue is on its card.
+
+
+
+
+
THE COUNTERS
+
+
COLLISIONS AVOIDED
grants that waited for a held card instead of loading on top of the holder
+
VRAM HANDED OFF
peak measured bytes of every lease whose memory the card then showed freed
+
DRAINS
holders asked to give the berth back
+
AGROUND CAUGHT
stuck releases caught before the card stayed dirty
+
STOWAWAYS CAUGHT
VRAM holders found on a card with no lease
+
CONTESTED
grants made while another holder shared the card
+
REFUSED
requests turned away when their wait ran out
+
JUGGLES STOPPED
handoffs between two engines at one priority that the ranking settled instead of flipping the card back and forth
+
+
Every count is an event the node saw. Berth never estimates what might have happened.
+
+
+
+
CONTEXT AND KV
+
Context residency draws each request's cached, fresh and generated tokens against the model's context window, so a request that would not fit is plain to see. The KV pool shows each engine's live slots (the llama.cpp family's /slots) against their windows, with marks at 75% and 90%. Longer ranges show the per-minute peak fill; the live slots stay live. An engine without these endpoints reports no window and no slots, and the panels say so.
+
+
+
+
GPU AND MACHINE
+
Each card reports power against its limit, temperature and utilisation, and in advanced mode its SM and memory clocks, fan and PCIe link. n/a means the driver did not report a value; it never means zero. The machine panel is the host: CPU, cores, memory, swap and load. Its socket line counts event-stream write timeouts and dropped slow readers, so a stalled log starts there.
+
+
+
+
THE LIBRARY
+
The library lists the model files on this machine, the same rows berth models prints: shelf, quant, size, and the CACHED share in the kernel page cache. COOL means half the file or more is cached, COLD less. The kernel takes those pages back when it needs the RAM, so CACHED predicts a faster reload and says nothing about RAM held. SERVED BY names the engine that has the model loaded, and EST VRAM is that engine's learned footprint, an estimate and never a reservation. The copy buttons give berth prefetch NAME and berth models show NAME.
+
+
+
+
CLAIMS, STOWAWAYS AND NO TICKET
+
A game that a hold claim matches holds the card under a claim:NAME lease, so engines drain and wait until it exits. berth hold does the same by hand. A track claim names an app's card memory (OBS, Blender) so it is not reported as a stowaway. The CLAIMS panel lists the rules matching now, as berth claims does, with the card on a machine with several, and counts the idle ones. A stowaway is any process holding VRAM without a lease: Berth reports it and never stops it. A NO TICKET run is an engine started while the node was down, so it holds the card outside any lease.
+
+
+
+
FROM THE TERMINAL
+
The harbor reads the node and changes nothing. Switching and stopping engines from here will need a paired device (berth pair), and that is not built yet. The gateway has its own page at /gui/ on the gateway's port, where an engine with a start command can be launched. Everything this page shows is also on the terminal:
+
berth status
berth events -f
-berth run
+berth claims
+berth hold --reason game
berth config check
diff --git a/internal/node/node.go b/internal/node/node.go
index db115ee..0aba0fd 100644
--- a/internal/node/node.go
+++ b/internal/node/node.go
@@ -107,14 +107,16 @@ func answers(name, addr, key string) bool {
}
// View is a node's status narrowed to the cards key picks, as Pick picks
-// them: their readings, leases, waiters and squatters. What holds every card
-// (claims, a passthrough, a probe error) stays. nil when st is nil or key
-// names a card st does not have.
+// them: their readings, leases, waiters and squatters. A claim stays listed,
+// active only when it is active on one of them (a claim with no cards, from
+// an older berthd, counts as on every card). What holds every card (a
+// passthrough, a probe error) stays. nil when st is nil or key names a card
+// st does not have.
func View(st *proto.Status, key string) *proto.Status {
if st == nil {
return nil
}
- cards := cardsOf(st, key)
+ cards := CardsOf(st, key)
if cards == nil {
return nil
}
@@ -124,6 +126,12 @@ func View(st *proto.Status, key string) *proto.Status {
v.Leases = slices.DeleteFunc(slices.Clone(st.Leases), func(l proto.Lease) bool { return !on(l.Card) })
v.Waiting = slices.DeleteFunc(slices.Clone(st.Waiting), func(l proto.Lease) bool { return !on(l.Card) })
v.Squatters = slices.DeleteFunc(slices.Clone(st.Squatters), func(s proto.Squatter) bool { return !on(s.Card) })
+ v.Claims = slices.Clone(st.Claims)
+ for i, cl := range v.Claims {
+ if len(cl.Cards) > 0 && !slices.ContainsFunc(cl.Cards, on) {
+ v.Claims[i].Active, v.Claims[i].Lease = false, ""
+ }
+ }
v.Passthrough = inVM(v.GPUs)
// A view of some cards carries only their probe errors. One that keeps
// every card keeps the status's own, which a berthd older than per-card
@@ -168,9 +176,9 @@ func inVM(gpus []proto.GPU) *proto.Passthrough {
return first
}
-// cardsOf is the names of the cards key picks in a node's status: "" is
+// CardsOf is the names of the cards key picks in a node's status: "" is
// the first card. nil when a name in key is no card in st.
-func cardsOf(st *proto.Status, key string) []string {
+func CardsOf(st *proto.Status, key string) []string {
var cards []proto.GPU
for _, g := range st.GPUs {
if g.Scheduled {
@@ -211,6 +219,21 @@ func (n *Node) Pick(key string) ([]*Card, error) {
return out, nil
}
+// Cordon stops (on) or resumes new grants on the cards key picks, as Pick
+// picks them, and returns their names.
+func (n *Node) Cordon(key string, on bool) ([]string, error) {
+ cards, err := n.Pick(key)
+ if err != nil {
+ return nil, err
+ }
+ names := make([]string, len(cards))
+ for i, c := range cards {
+ c.B.Cordon(on)
+ names[i] = c.Name
+ }
+ return names, nil
+}
+
// Stop drains every lease of engine on every card.
func (n *Node) Stop(engine string) ([]string, error) {
var ids []string
diff --git a/internal/node/node_test.go b/internal/node/node_test.go
index 9e07a2b..7c4d26e 100644
--- a/internal/node/node_test.go
+++ b/internal/node/node_test.go
@@ -260,7 +260,8 @@ func TestViewIsTheCardsAKeyNames(t *testing.T) {
Leases: []proto.Lease{{Engine: "chat", Card: "gpu0"}, {Engine: "maya", Card: "gpu1"}},
Waiting: []proto.Lease{{Engine: "voice", Card: "gpu1"}},
Squatters: []proto.Squatter{{PID: 7, Card: "gpu0"}},
- Claims: []proto.Claim{{Name: "steam-game", Active: true}},
+ Claims: []proto.Claim{{Name: "steam-game", Active: true}, {Name: "sim", Active: true, Lease: "l-4", Cards: []string{"gpu1"}},
+ {Name: "blender", Action: "track"}},
}
engines := func(ls []proto.Lease) string {
var out []string
@@ -270,14 +271,14 @@ func TestViewIsTheCardsAKeyNames(t *testing.T) {
return strings.Join(out, ",")
}
for _, tc := range []struct {
- key, gpus, leases, waiting string
- squatters int
+ key, gpus, leases, waiting, active string
+ squatters int
}{
- {"", "gpu0", "chat", "", 1},
- {"gpu1", "gpu1", "maya", "voice", 0},
- {"0000:09:00.0", "gpu1", "maya", "voice", 0},
- {"00000000:09:00.0", "gpu1", "maya", "voice", 0},
- {"gpu1,0000:01:00.0", "gpu0,gpu1", "chat,maya", "voice", 1},
+ {"", "gpu0", "chat", "", "steam-game", 1},
+ {"gpu1", "gpu1", "maya", "voice", "steam-game,sim", 0},
+ {"0000:09:00.0", "gpu1", "maya", "voice", "steam-game,sim", 0},
+ {"00000000:09:00.0", "gpu1", "maya", "voice", "steam-game,sim", 0},
+ {"gpu1,0000:01:00.0", "gpu0,gpu1", "chat,maya", "voice", "steam-game,sim", 1},
} {
v := View(st, tc.key)
var gpus []string
@@ -296,14 +297,20 @@ func TestViewIsTheCardsAKeyNames(t *testing.T) {
if len(v.Squatters) != tc.squatters {
t.Errorf("%q: %d squatters, want %d", tc.key, len(v.Squatters), tc.squatters)
}
- if len(v.Claims) != 1 {
- t.Errorf("%q: claims %v, want the active claim kept", tc.key, v.Claims)
+ var active []string
+ for _, cl := range v.Claims {
+ if cl.Active {
+ active = append(active, cl.Name)
+ }
+ }
+ if got := strings.Join(active, ","); len(v.Claims) != 3 || got != tc.active {
+ t.Errorf("%q: active claims %s of %d, want %s of 3", tc.key, got, len(v.Claims), tc.active)
}
}
if v := View(st, "gpu9"); v != nil {
t.Errorf("gpu9: %+v, want nil for a card the node lacks", v)
}
- if len(st.Leases) != 2 || len(st.Squatters) != 1 {
+ if len(st.Leases) != 2 || len(st.Squatters) != 1 || !st.Claims[1].Active {
t.Errorf("View changed the status it read: %+v", st)
}
}
diff --git a/internal/proto/proto.go b/internal/proto/proto.go
index 0d5e547..17ca6d3 100644
--- a/internal/proto/proto.go
+++ b/internal/proto/proto.go
@@ -18,8 +18,9 @@ import (
const (
OpAcquire = "acquire" // client: ask for a berth; blocks until granted or refused
OpAttach = "attach" // client: the engine is running as this PID
- // OpFinished: client: the engine exited 0, so it stopped answering on
- // purpose. Sent just before the hang-up that ends the lease.
+ // OpFinished: client: the engine stopped answering on purpose: it exited
+ // 0, or the run ended it (a drain, a failed wake, a stop signal it
+ // forwarded). Sent just before the hang-up that ends the lease.
OpFinished = "finished"
// OpReattach: client: an engine already running under berth run (PID plus
// PIDStart) lost its daemon and asks the new one to lease it the card again.
@@ -45,6 +46,12 @@ const (
// higher-priority waiter would. The server answers once with the leases
// it asked.
OpStop = "stop"
+ // OpCordon, OpUncordon: client: stop or resume new grants on the cards
+ // Card names (as OpAcquire names them). A cordoned card keeps its
+ // holders and still grants a hold. The server answers with the same op
+ // and Cards, the cards' names, or OpRefused for a card the node lacks.
+ OpCordon = "cordon"
+ OpUncordon = "uncordon"
// OpPark: the runner holding a draining lease asks to park its engine
// in RAM; the daemon answers OpParkOK or OpRefused with the why line.
// OpParked: the runner's park call is done; the daemon watches for the
@@ -145,7 +152,9 @@ type Msg struct {
Container string `json:"container,omitempty"`
Lease string `json:"lease,omitempty"`
// OpStop: Leases is the reply, the leases the daemon asked to drain.
- Leases []string `json:"leases,omitempty"`
+ Leases []string `json:"leases,omitempty"`
+ // OpCordon, OpUncordon: Cards is the reply, the cards' names.
+ Cards []string `json:"cards,omitempty"`
Reason string `json:"reason,omitempty"`
Status *Status `json:"status,omitempty"`
Since uint64 `json:"since,omitempty"` // OpEvents: only events with a higher Seq
@@ -238,6 +247,8 @@ const (
EvResumed = "lease.resumed" // a parked engine was granted the card back
EvUnparked = "lease.unparked" // a parked engine loaded back onto the card with no resume; it is active again
EvParkRefused = "lease.park_refused" // the RAM could not take the park; the drain runs as before
+ EvCordoned = "card.cordoned" // berth cordon: no new grants on the card but holds
+ EvUncordoned = "card.uncordoned" // berth uncordon: the card grants again
EvParkEvicted = "lease.park_evicted" // the parked engine is being stopped: ttl, RAM floor or pressure
EvLending = "lease.lending" // a serving holder was asked to give part of its card to a waiter
EvLent = "lease.lent" // the engine gave it, and a card read after the answer shows it free
@@ -421,6 +432,8 @@ type GPU struct {
// Passthrough is this card's VM state while it does not serve; nil
// when the card is normal.
Passthrough *Passthrough `json:"passthrough,omitempty"`
+ // Cordoned: berth cordon stopped new grants on this card, holds aside.
+ Cordoned bool `json:"cordoned,omitempty"`
// Telemetry as the driver reports it; -1 = not reported.
Util float64 `json:"util_pct"`
diff --git a/internal/runner/chaos_test.go b/internal/runner/chaos_test.go
index 013cb21..6e83c99 100644
--- a/internal/runner/chaos_test.go
+++ b/internal/runner/chaos_test.go
@@ -92,7 +92,7 @@ func TestStopSignalKillsCheckpointedEngine(t *testing.T) {
hog, died := startStubborn(t)
done := make(chan struct{})
defer close(done)
- go forwardSignals(hog, done, func() bool { return true })
+ go forwardSignals(hog, done, func() bool { return true }, func() {})
time.Sleep(50 * time.Millisecond)
if err := syscall.Kill(syscall.Getpid(), syscall.SIGTERM); err != nil {
t.Fatal(err)
diff --git a/internal/runner/finished_test.go b/internal/runner/finished_test.go
index 582fe71..57fb176 100644
--- a/internal/runner/finished_test.go
+++ b/internal/runner/finished_test.go
@@ -12,8 +12,10 @@ import (
"log/slog"
"net/http"
"net/http/httptest"
+ "os"
"path/filepath"
"sync/atomic"
+ "syscall"
"testing"
"time"
@@ -26,17 +28,32 @@ import (
)
// A batch engine that exits 0 stopped answering on purpose: the engines
-// tab shows it down, not latched (issue 265). One that exits non-zero
-// keeps the latch, so a crash still reads as one.
+// tab shows it down, not latched (issue 265). So does one Berth stopped,
+// through berth stop or a stop signal from its supervisor, whatever code
+// the signal leaves it. One that exits non-zero on its own keeps the
+// latch, so a crash still reads as one.
func TestACleanExitDoesNotLatchTheEngine(t *testing.T) {
+ forever := []string{"sh", "-c", "exec sleep 30"}
for _, c := range []struct {
- exit int
+ name string
+ argv []string
+ stop func(t *testing.T, b *broker.Broker)
want string
}{
- {0, "down"},
- {1, "latched"},
+ {"exit 0", []string{"sh", "-c", "exit 0"}, nil, "down"},
+ {"exit 1", []string{"sh", "-c", "exit 1"}, nil, "latched"},
+ {"berth stop", forever, func(t *testing.T, b *broker.Broker) {
+ if _, err := b.Stop("deduper"); err != nil {
+ t.Fatal(err)
+ }
+ }, "down"},
+ {"supervisor stop", forever, func(t *testing.T, _ *broker.Broker) {
+ if err := syscall.Kill(os.Getpid(), syscall.SIGTERM); err != nil {
+ t.Fatal(err)
+ }
+ }, "down"},
} {
- t.Run(fmt.Sprintf("exit %d", c.exit), func(t *testing.T) {
+ t.Run(c.name, func(t *testing.T) {
var up atomic.Bool
up.Store(true)
eng := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -66,10 +83,19 @@ func TestACleanExitDoesNotLatchTheEngine(t *testing.T) {
if s := w.Snapshot()[0].State; s != "up" {
t.Fatalf("before the run: state %q, want up", s)
}
- code := Run(Options{Socket: sock, Engine: "deduper", Priority: "batch", Grace: time.Second,
- Argv: []string{"sh", "-c", fmt.Sprintf("exit %d", c.exit)}, Stderr: io.Discard})
- if code != c.exit {
- t.Fatalf("run exit %d, want %d", code, c.exit)
+ ran := make(chan int, 1)
+ go func() {
+ ran <- Run(Options{Socket: sock, Engine: "deduper", Priority: "batch", Grace: time.Second,
+ Argv: c.argv, Stderr: io.Discard})
+ }()
+ if c.stop != nil {
+ waitFor(t, "the lease granted", func() bool { return len(b.Status().Leases) == 1 })
+ c.stop(t, b)
+ }
+ select {
+ case <-ran:
+ case <-time.After(10 * time.Second):
+ t.Fatal("the run never ended")
}
up.Store(false)
waitFor(t, "the lease released", func() bool { return len(b.Status().Leases) == 0 })
diff --git a/internal/runner/options.go b/internal/runner/options.go
index 6d9fe48..b897fe4 100644
--- a/internal/runner/options.go
+++ b/internal/runner/options.go
@@ -66,7 +66,8 @@ func OptionsFor(e *config.Engine, o Over) Options {
APIKeyEnv: e.APIKeyEnv, APIKeyFile: e.APIKeyFile,
ExitComfyUIIdle: e.ExitComfyUIIdle, ExitComfyUIIdleAfter: e.ExitComfyUIIdleAfter.Duration,
Grace: grace, FailClosed: o.FailClosed,
- Argv: argv, Stderr: o.Stderr, Output: o.Output, Log: o.Log,
+ Slice: e.Slice,
+ Argv: argv, Stderr: o.Stderr, Output: o.Output, Log: o.Log,
}
// kv_snapshots: the drain's snapshot step. The slot calls go to the
// entry's drain_slots base when it names one, else the drain_post
diff --git a/internal/runner/park.go b/internal/runner/park.go
index 11afdf3..5687323 100644
--- a/internal/runner/park.go
+++ b/internal/runner/park.go
@@ -43,6 +43,10 @@ type leaseLine struct {
conn *proto.Conn
replies chan proto.Msg
parked bool
+ // stopped is set once the run itself sets out to end the engine (a
+ // drain past the park, a failed wake, a forwarded stop signal), so the
+ // exit that follows is on purpose whatever its code.
+ stopped bool
probed *parkVerdict
lends map[string]lend.Ticket
// the 503 watch's pacing: how long it reads the engine's ledger and how
@@ -67,6 +71,18 @@ func (l *leaseLine) isParked() bool {
return l.parked
}
+func (l *leaseLine) askStop() {
+ l.mu.Lock()
+ l.stopped = true
+ l.mu.Unlock()
+}
+
+func (l *leaseLine) stopWasAsked() bool {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.stopped
+}
+
func (l *leaseLine) setConn(conn *proto.Conn) {
l.mu.Lock()
l.conn = conn
@@ -234,13 +250,13 @@ func (l *leaseLine) resume(o Options, cmd *exec.Cmd, done <-chan struct{}, say f
cancel()
if err != nil {
say("resuming %s failed (%v); stopping it", o.Engine, err)
- stopNow(o, cmd, done, say)
+ l.stopNow(o, cmd, done, say)
return
}
}
if !o.parkReady() {
say("%s never answered after the wake; stopping it", o.Engine)
- stopNow(o, cmd, done, say)
+ l.stopNow(o, cmd, done, say)
return
}
if o.front != nil {
@@ -289,7 +305,8 @@ func askReady(url, key string) bool {
// stopNow runs the stop ladder's tail: terminate, then kill after Grace.
// The engine exits, the wait returns, and the lease is released with it.
-func stopNow(o Options, cmd *exec.Cmd, done <-chan struct{}, say func(string, ...any)) {
+func (l *leaseLine) stopNow(o Options, cmd *exec.Cmd, done <-chan struct{}, say func(string, ...any)) {
+ l.askStop()
terminate(cmd)
select {
case <-done:
diff --git a/internal/runner/runner.go b/internal/runner/runner.go
index 4cbc9bd..f0915d5 100644
--- a/internal/runner/runner.go
+++ b/internal/runner/runner.go
@@ -146,6 +146,12 @@ type Options struct {
// nothing else is ever stopped. Container and Compose are mutually
// exclusive, and config.Validate keeps them so.
Compose string
+ // Slice is the systemd slice (NAME.slice) the engine's scope is
+ // created under (berth run --slice, the entry's slice): the slice
+ // unit carries resource policy, CPU, memory and swap limits, so the
+ // engine and everything it forks inherit them. Linux user manager
+ // only; elsewhere the engine runs where the run leaves it.
+ Slice string
// CLI is the runtime CLI exec: every docker or podman inspect, list and
// stop runs through it, under the timeout the caller names, and it
// returns the command's combined output; nil = the real CLI, tests fake
@@ -268,7 +274,7 @@ func Run(o Options) int {
// line is the lease's live conn and park state; the drain, the resume
// and the container resolver ride it across reattach loops.
line := newLine(conn)
- go forwardSignals(cmd, done, func() bool { return o.Checkpoint && line.isParked() })
+ go forwardSignals(cmd, done, func() bool { return o.Checkpoint && line.isParked() }, line.askStop)
// A container engine resolves its containers to their full IDs and names
// them on the lease, so the broker attributes the engine's GPU processes
@@ -389,7 +395,7 @@ func Run(o Options) int {
err = nil // exited cleanly; a grandchild still held the output copy
}
code := exitCode(cmd, err)
- if code == 0 {
+ if code == 0 || line.stopWasAsked() {
_ = line.send(proto.Msg{Op: proto.OpFinished})
}
close(done)
@@ -706,7 +712,7 @@ func runUnleased(o Options, reason string) int {
afterStart(cmd)
defer record(o, cmd, reason)()
done := make(chan struct{})
- go forwardSignals(cmd, done, func() bool { return false })
+ go forwardSignals(cmd, done, func() bool { return false }, func() {})
err := cmd.Wait()
afterWait(cmd)
close(done)
@@ -750,7 +756,7 @@ func visibleDevices(card string, granted proto.Msg, env func(string) (string, bo
}
func command(o Options, out io.Writer) *exec.Cmd {
- argv := scopeToChild(o.Engine, o.Argv, o.Log)
+ argv := scopeToChild(o.Engine, o.Slice, o.Argv, o.Log)
cmd := exec.Command(argv[0], argv[1:]...)
cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
if out != nil {
@@ -771,7 +777,8 @@ func ExitCode(cmd *exec.Cmd, err error) int { return exitCode(cmd, err) }
// to the engine's whole process group. A checkpointed engine is killed
// instead: its CUDA calls block until a restore, so its own shutdown never
// finishes, and the supervisor kills this wrapper before a grace would end.
-func forwardSignals(cmd *exec.Cmd, done <-chan struct{}, locked func() bool) {
+// asked records each one as a stop on purpose.
+func forwardSignals(cmd *exec.Cmd, done <-chan struct{}, locked func() bool, asked func()) {
ch := make(chan os.Signal, 4)
signal.Notify(ch, stopSignals...)
defer signal.Stop(ch)
@@ -780,6 +787,7 @@ func forwardSignals(cmd *exec.Cmd, done <-chan struct{}, locked func() bool) {
case <-done:
return
case s := <-ch:
+ asked()
if locked() {
kill(cmd)
continue
@@ -869,6 +877,7 @@ func drain(o Options, cmd *exec.Cmd, line *leaseLine, done <-chan struct{}, reas
if o.Park != nil || o.Hook {
if line.isParked() && o.Checkpoint {
say("killing checkpointed %s: it cannot shut down while its CUDA calls are locked", o.Engine)
+ line.askStop()
kill(cmd)
return
}
@@ -878,6 +887,8 @@ func drain(o Options, cmd *exec.Cmd, line *leaseLine, done <-chan struct{}, reas
return
}
}
+ // From here every step ends the engine.
+ line.askStop()
if o.DrainPost != "" {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, o.DrainPost, strings.NewReader(""))
diff --git a/internal/runner/runner_unix.go b/internal/runner/runner_unix.go
index 92c598b..a82812f 100644
--- a/internal/runner/runner_unix.go
+++ b/internal/runner/runner_unix.go
@@ -38,24 +38,42 @@ func afterWait(*exec.Cmd) {}
// `berth run` inside another lease's engine, as llama-swap runs its
// models): two leases never share a scope. Without a user manager the
// child runs in this cgroup, as always.
-func scopeToChild(as string, argv []string, log *slog.Logger) []string {
+func scopeToChild(as, slice string, argv []string, log *slog.Logger) []string {
// Tests start fake engines: never in the developer's own user manager.
if testing.Testing() {
return argv
}
- if !scopeAllowed(os.Getenv) {
- if log != nil {
- log.Debug("inside a Type=notify service; the engine stays in its cgroup", "engine", as)
- }
+ return pickScopeArgv(scopeAllowed(os.Getenv), sysd.ScopeReady(), as, slice, argv, log)
+}
+
+// pickScopeArgv places the engine in the scope. An operator who set a
+// slice for its limits gets an engine that can swap when the scope is
+// dropped, so a slice is never dropped quietly: the Warn names the
+// engine, the slice and the reason. A plain run with no slice reads as
+// always, at Debug.
+func pickScopeArgv(allowed, ready bool, as, slice string, argv []string, log *slog.Logger) []string {
+ if !allowed {
+ logScopeDrop(log, as, slice, "inside a Type=notify service; the engine stays in its cgroup")
return argv
}
- if !sysd.ScopeReady() {
- if log != nil {
- log.Debug("no systemd user manager; the engine runs in this run's cgroup", "engine", as)
- }
+ if !ready {
+ logScopeDrop(log, as, slice, "no systemd user manager; the engine runs in this run's cgroup")
return argv
}
- return sysd.ScopeArgv("berth-run", as, argv)
+ return sysd.ScopeArgv("berth-run", as, slice, argv)
+}
+
+// logScopeDrop tells the operator why the engine keeps the caller's
+// cgroup, at Warn when a slice made that placement matter.
+func logScopeDrop(log *slog.Logger, as, slice, reason string) {
+ if log == nil {
+ return
+ }
+ if slice == "" {
+ log.Debug(reason, "engine", as)
+ return
+ }
+ log.Warn(reason+"; the slice is dropped", "engine", as, "slice", slice)
}
// scopeAllowed is false inside a Type=notify service (NOTIFY_SOCKET set):
diff --git a/internal/runner/runner_windows.go b/internal/runner/runner_windows.go
index 81fd63e..c50b585 100644
--- a/internal/runner/runner_windows.go
+++ b/internal/runner/runner_windows.go
@@ -35,7 +35,7 @@ func isolate(cmd *exec.Cmd) {
// scopeToChild has no user manager to answer to on this platform; Job
// objects and the attribution they allow come in a later ticket.
-func scopeToChild(as string, argv []string, log *slog.Logger) []string { return argv }
+func scopeToChild(as, slice string, argv []string, log *slog.Logger) []string { return argv }
// afterStart puts the engine in a job object. A child it spawns in the first
// instant before this lands escapes the job; the drain still reaches the
diff --git a/internal/runner/scope_unix_test.go b/internal/runner/scope_unix_test.go
index 4656f01..d6882d2 100644
--- a/internal/runner/scope_unix_test.go
+++ b/internal/runner/scope_unix_test.go
@@ -5,7 +5,13 @@
package runner
-import "testing"
+import (
+ "bytes"
+ "encoding/json"
+ "log/slog"
+ "strings"
+ "testing"
+)
func TestScopeAllowed(t *testing.T) {
for _, tc := range []struct {
@@ -27,3 +33,56 @@ func TestScopeAllowed(t *testing.T) {
}
}
}
+
+// An operator who set slice for its limits gets an engine that can swap
+// when the scope is dropped, so a slice is never dropped quietly: the
+// line names the engine, the slice and the reason, at Warn. A plain run
+// with no slice keeps the Debug line it always had.
+func TestScopeDropWarnsTheSlice(t *testing.T) {
+ argv := []string{"engine", "--flag"}
+ for _, tc := range []struct {
+ name string
+ allowed bool
+ ready bool
+ slice string
+ want string // level to expect; "" = no line at all
+ }{
+ {"notify service, slice", false, true, "strata.slice", "WARN"},
+ {"no user manager, slice", true, false, "strata.slice", "WARN"},
+ {"notify service, no slice", false, true, "", "DEBUG"},
+ {"no user manager, no slice", true, false, "", "DEBUG"},
+ {"scope taken", true, true, "strata.slice", ""},
+ } {
+ var buf bytes.Buffer
+ log := slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
+ got := pickScopeArgv(tc.allowed, tc.ready, "fixture", tc.slice, argv, log)
+ if strings.Join(got, " ") != strings.Join(argv, " ") {
+ t.Errorf("%s: the engine's argv changed: %q", tc.name, got)
+ }
+ if tc.want == "" {
+ if buf.Len() > 0 {
+ t.Errorf("%s: a placed scope logged: %s", tc.name, buf.String())
+ }
+ continue
+ }
+ var rec map[string]any
+ if err := json.Unmarshal(buf.Bytes(), &rec); err != nil {
+ t.Fatalf("%s: no log line: %v (%q)", tc.name, err, buf.String())
+ }
+ if rec["level"] != tc.want {
+ t.Errorf("%s: level = %v, want %s (%q)", tc.name, rec["level"], tc.want, buf.String())
+ }
+ if rec["engine"] != "fixture" {
+ t.Errorf("%s: the line lost the engine: %q", tc.name, buf.String())
+ }
+ if tc.slice == "" {
+ continue
+ }
+ if rec["slice"] != tc.slice {
+ t.Errorf("%s: the line lost the slice: %q", tc.name, buf.String())
+ }
+ if msg, _ := rec["msg"].(string); !strings.Contains(msg, "slice is dropped") {
+ t.Errorf("%s: the line does not name the drop: %q", tc.name, msg)
+ }
+ }
+}
diff --git a/internal/server/cordon.go b/internal/server/cordon.go
new file mode 100644
index 0000000..82ce820
--- /dev/null
+++ b/internal/server/cordon.go
@@ -0,0 +1,27 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+// OpCordon and OpUncordon. The named cards stop or resume new grants; the
+// reply names them. A cordon drains nothing.
+
+import (
+ "log/slog"
+
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+func cordon(conn *proto.Conn, log *slog.Logger, req proto.Msg, n *node.Node) {
+ if req.Card == "" {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: "name the card to " + req.Op})
+ return
+ }
+ cards, err := n.Cordon(req.Card, req.Op == proto.OpCordon)
+ if err != nil {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: err.Error()})
+ return
+ }
+ reply(conn, log, proto.Msg{Op: req.Op, Cards: cards})
+}
diff --git a/internal/server/server.go b/internal/server/server.go
index 953d577..755561c 100644
--- a/internal/server/server.go
+++ b/internal/server/server.go
@@ -115,8 +115,8 @@ type Extras struct {
Alerts func() *proto.Alerts
// Pair is the harbor's device pairing; nil turns the OpPair* ops off.
Pair *pair.Store
- // Finished hears that a run's engine exited 0, by the lease's engine
- // name.
+ // Finished hears that a run's engine ended on purpose (exit 0, or a
+ // stop the run carried out), by the lease's engine name.
Finished func(engine string)
}
@@ -186,6 +186,8 @@ func handle(ctx context.Context, c net.Conn, n *node.Node, x Extras, log *slog.L
reclaim(conn, log, first, n)
case proto.OpStop:
stop(conn, log, first, n)
+ case proto.OpCordon, proto.OpUncordon:
+ cordon(conn, log, first, n)
case proto.OpPairCode, proto.OpPairRedeem, proto.OpPairCheck, proto.OpPairList, proto.OpPairRevoke:
pairing(conn, log, first, x.Pair)
case proto.OpResume:
diff --git a/internal/skill/SKILL.md b/internal/skill/SKILL.md
index afcadbc..cc07665 100644
--- a/internal/skill/SKILL.md
+++ b/internal/skill/SKILL.md
@@ -45,13 +45,16 @@ do not assume any engine is protected.
| Run a configured engine without retyping its command | `berth run --as NAME` with no command runs the `start` of NAME's `[[engine]]` entry |
| Run an app that must not be drained (Blender, a training job) | `berth run --as NAME --pinned -- CMD` |
| Give an engine a minimum turn before a request can drain it | `berth run --as NAME --min-turn 5m -- CMD`, or `min_turn = "5m"` in its engine entry (a hold still wins; `berth status` shows the turn left in the TURN column) |
-| Free the card for the user (game, call) | `berth hold [--for 2h] [--reason game]` (ends on Ctrl-C) |
+| Free the card for the user (game, call) | `berth hold [--for 2h] [--reason game] [--gpu gpu1]` (ends on Ctrl-C; default the first card) |
| Call a model and let Berth start its engine | the gateway: `http://127.0.0.1:9799/v1`, bearer key from `berth key` |
| Wrap every llama-swap model at once | `berth wire llama-swap --config CFG` (plan), `--write` (apply, keeps a backup), `--write --update` (rewrite wire's flags on lines it wrote, keeping flags added by hand), `--wait 10m` (every model waits that long for the card) |
| Run an engine that lives in Docker or Podman | `berth run --as NAME --container CNAME -- docker run --rm --name CNAME ...` |
| Run a Docker Compose project as one engine | `berth run --as NAME --compose /srv/APP/compose.yaml -- docker compose -f /srv/APP/compose.yaml up` (the lease covers every container of the project) |
+| Apply a systemd slice's CPU/memory/swap policy to the engine itself | `berth run --as NAME --slice NAME.slice -- CMD`, or `slice = "NAME.slice"` in its engine entry (Linux user manager) |
| Name an engine that discovery does not find | an `[[engine]]` entry in `berth.toml` with `type`, `url` and, for the gateway, `start` and `serves`. A Strata entry needs `type = "strata"` (and `whole_machine = true`), or its requests never reach the ledger |
+| Call a model on one card of several | the card's base URL: `http://127.0.0.1:9799/cards/gpu1/v1` (same key and model names) |
| Warm an engine before its first request | `GET /v1/berth/wait?model=M` on the gateway (blocks until it answers) |
+| Close a card to new work while the others serve | `berth cordon gpu1` (holders keep running; `berth uncordon gpu1` reopens it) |
| Stop an engine Berth started | `berth stop ENGINE [--wait]` (normal drain, parked engines included; engines outlive a gateway restart) |
| Bring a parked engine back | `berth resume ENGINE` (or just call its model through the gateway) |
| Read a model's file ahead into RAM | `berth prefetch MODEL` (page cache only; nothing is reserved) |
@@ -101,6 +104,11 @@ Rules for a workflow:
- Lease states: `waiting` (with its place in line), `active` (HOT), `draining`, `parking`, `parked` (WARM: the
engine moved its weights to RAM and holds only a small residual; `berth resume` or a request brings it back),
`releasing`, `stuck` (its memory never left: see below).
+- The UP column in `berth status`: `yes` answers. `LATCHED` answered before and stopped without its run ending it:
+ a crash or a hang worth reporting. `NO` never answered here, or ended on purpose (exit 0, or a stop Berth carried
+ out).
+- `berth stop` and `POST /v1/berth/stop` act on leases. An engine with no running lease is refused with `no running
+ engine named NAME under a lease` (409 over HTTP): there is nothing to stop, and the next request starts it again.
- `berth status` shows EST. VRAM per engine: the peak its last run measured. An estimate, never a reservation.
- `berth models` shows CACHED per model: how much of its file the kernel still holds in RAM (COOL) or not (COLD).
- A streaming request to the gateway gets `: berth: loading ...` comment lines while its engine starts; that is
@@ -120,6 +128,7 @@ name = "training"
cmdline = 'train\.py' # regex on the command line (Windows: the exe's full path)
action = "hold" # hold: drain engines while it runs; track: just name its memory
linger = "30s"
+gpu = "gpu1" # hold only this card on a node with several; default every card
[claims]
builtin = true # false turns the built-in pack off
diff --git a/internal/sysd/scope_slice_unix_test.go b/internal/sysd/scope_slice_unix_test.go
new file mode 100644
index 0000000..711f376
--- /dev/null
+++ b/internal/sysd/scope_slice_unix_test.go
@@ -0,0 +1,25 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package sysd
+
+import (
+ "strings"
+ "testing"
+)
+
+// slice placement is the whole point: the engine scope lands under the
+// operator's slice, so the slice unit's resource policy covers it.
+func TestScopeArgvPlacesTheScopeUnderTheSlice(t *testing.T) {
+ got := scopeArgv("berth-run-strata-0000ffff", "strata.slice", []string{"run.sh"})
+ want := []string{"systemd-run", "--user", "--scope", "--unit", "berth-run-strata-0000ffff", "--collect", "--quiet", "--slice", "strata.slice", "--", "run.sh"}
+ if strings.Join(got, " ") != strings.Join(want, " ") {
+ t.Errorf("argv = %q, want %q", got, want)
+ }
+ bare := scopeArgv("berth-run-strata-0000ffff", "", []string{"run.sh"})
+ if strings.Contains(strings.Join(bare, " "), "--slice") {
+ t.Errorf("a run with no slice gained one: %q", bare)
+ }
+}
diff --git a/internal/sysd/sysd.go b/internal/sysd/sysd.go
index bd50fec..c012c26 100644
--- a/internal/sysd/sysd.go
+++ b/internal/sysd/sysd.go
@@ -8,6 +8,30 @@
// processes for attribution even when they escape the process tree.
package sysd
+import "fmt"
+
+// CheckSlice accepts the slice name ScopeArgv passes to systemd-run: a
+// unit name ending in .slice, carrying only what unit names take (ASCII
+// letters, digits, and . - _ : @). systemd-run takes the name verbatim;
+// a run that would fail its scope start says so at once, while the
+// engine is still untouched.
+func CheckSlice(name string) error {
+ const suffix = ".slice"
+ if len(name) <= len(suffix) || name[len(name)-len(suffix):] != suffix {
+ return fmt.Errorf("slice %q must name a .slice unit, such as app-berth.slice", name)
+ }
+ for i := 0; i < len(name)-len(suffix); i++ {
+ c := name[i]
+ switch {
+ case c >= 'A' && c <= 'Z', c >= 'a' && c <= 'z', c >= '0' && c <= '9',
+ c == '.', c == '_', c == '-', c == ':', c == '@':
+ default:
+ return fmt.Errorf("slice %q has byte %q, which no unit name takes", name, c)
+ }
+ }
+ return nil
+}
+
// UnitName maps an engine name onto what a systemd unit name may carry
// (ASCII letters, digits, and . - _ :), replacing every other byte with
// _ and capping the length so a long name still leaves room for the
diff --git a/internal/sysd/sysd_test.go b/internal/sysd/sysd_test.go
index a045e19..364b0fe 100644
--- a/internal/sysd/sysd_test.go
+++ b/internal/sysd/sysd_test.go
@@ -30,3 +30,18 @@ func TestUnitName(t *testing.T) {
t.Errorf("UnitName length = %d, want at most 64", len(got))
}
}
+
+// A slice name reaches systemd-run verbatim, so it is validated here
+// while the run can still refuse and the engine stays untouched.
+func TestCheckSlice(t *testing.T) {
+ for _, ok := range []string{"strata.slice", "app-berth.slice", "foo@bar.slice", "a.slice", "x-1_2:.slice"} {
+ if err := CheckSlice(ok); err != nil {
+ t.Errorf("CheckSlice(%q) = %v, want nil", ok, err)
+ }
+ }
+ for _, bad := range []string{"", ".slice", "strata", "a b.slice", "a/b.slice", "app.slice/x", "strata.SLICE"} {
+ if err := CheckSlice(bad); err == nil {
+ t.Errorf("CheckSlice(%q) = nil, want an error", bad)
+ }
+ }
+}
diff --git a/internal/sysd/sysd_unix.go b/internal/sysd/sysd_unix.go
index 14bb24c..e83bed1 100644
--- a/internal/sysd/sysd_unix.go
+++ b/internal/sysd/sysd_unix.go
@@ -35,13 +35,26 @@ func ScopeReady() bool {
// --, --collect drops the scope when the run ends,
// --quiet keeps systemd-run's own notice out of the output. With --scope
// systemd-run execs the command in place, so its PID is unchanged.
-func ScopeArgv(prefix, name string, argv []string) []string {
+// slice, when set, is a validated .slice unit name: the scope is placed
+// under it, so the slice unit's resource policy (CPU, memory and swap
+// limits) applies to the engine and everything it forks.
+func ScopeArgv(prefix, name, slice string, argv []string) []string {
// Tests start fake engines: never in the developer's own user manager.
if testing.Testing() || !ScopeReady() {
return argv
}
var rnd [4]byte
_, _ = rand.Read(rnd[:])
- unit := fmt.Sprintf("%s-%s-%x", prefix, UnitName(name), rnd)
- return append([]string{"systemd-run", "--user", "--scope", "--unit", unit, "--collect", "--quiet", "--"}, argv...)
+ return scopeArgv(fmt.Sprintf("%s-%s-%x", prefix, UnitName(name), rnd), slice, argv)
+}
+
+// scopeArgv is the argv builder ScopeArgv runs when a manager is
+// reachable: tests drive it directly, so the shape (and the slice
+// placement) is asserted without touching a real manager.
+func scopeArgv(unit, slice string, argv []string) []string {
+ args := []string{"systemd-run", "--user", "--scope", "--unit", unit, "--collect", "--quiet"}
+ if slice != "" {
+ args = append(args, "--slice", slice)
+ }
+ return append(args, append([]string{"--"}, argv...)...)
}
diff --git a/internal/sysd/sysd_windows.go b/internal/sysd/sysd_windows.go
index a785141..73fc2ea 100644
--- a/internal/sysd/sysd_windows.go
+++ b/internal/sysd/sysd_windows.go
@@ -9,4 +9,4 @@ package sysd
func ScopeReady() bool { return false }
// ScopeArgv has no user manager to answer to on this platform.
-func ScopeArgv(prefix, name string, argv []string) []string { return argv }
+func ScopeArgv(prefix, name, slice string, argv []string) []string { return argv }