Skip to content

feat: berth cordon and uncordon a card - #281

Merged
T-Crypt merged 3 commits into
nextfrom
T-Crypt/cordon-card
Oct 31, 2026
Merged

T-Crypt merged 3 commits into
nextfrom
T-Crypt/cordon-card

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Third slice of v0.5: per-card gateway endpoints.

What changes for users

On a node with several cards, berth cordon gpu1 closes a card to new work while the other cards keep serving: for maintenance, a driver test, or before a VM takes the card. berth uncordon gpu1 reopens it. The card can be named by its name or its PCI address.

  • What holds the card keeps running. A cordon drains nothing, and berth stop ENGINE does that.
  • New leases wait, and their reason says the card is cordoned. A hold (berth hold, a game claim) is still granted.
  • berth status marks the card CORDONED, and berth events records card.cordoned and card.uncordoned.
  • The gateway treats the card as out of reach. A plain /v1 request goes to another card that serves its model (feat(gateway): place a plain request on the cheapest card #280's ranking), no background engine starts for the card, and a ComfyUI job for it queues.
  • The cordon lives in berthd's memory, so a berthd restart clears it. The docs and the command's help say so.

How

  • Broker: a cordoned flag. evaluate returns early for any waiter below Hold, Cordon(on) emits the event and schedules at once, and Status sets GPU.Cordoned.
  • Protocol: OpCordon and OpUncordon with Card, answered with the cards' names in Cards. node.Cordon uses Pick, so a comma list or a PCI address works as it does for berth run --gpu.
  • CLI: berth cordon and berth uncordon, modelled on berth stop, with JSON and YAML output.
  • Gateway: cardFree and outOfReach check a cordoned card. reach (from feat(gateway): place a plain request on the cheapest card #280) builds on both, so a cordoned card ranks last.

The branch merges next after #280. That merge commit also resolves the roadmap conflict, adds a "gpu0 cordoned" case to the #280 placement test, and adds the cordon to the gateway page's placement list.

Proof

  • TestCordonStopsNewGrantsUntilUncordoned (broker): a foreground waiter is kept with a reason naming the cordon, a hold is granted, and the waiter is granted after the uncordon. Status shows the flag both ways.
  • TestCordonAgainstADaemon (CLI): cordon and uncordon through the socket, CORDONED in berth status, the JSON flag, and exit 1 for an unknown card.
  • TestCordonedCardIsNotFree (gateway): outOfReach names the cordon on gpu1 and nothing on gpu0, and no background run is queued for gpu1 until the uncordon. With the two gateway changes reverted, both assertions fail.
  • The placement test's "gpu0 cordoned" case goes to gpu1.
  • Build, gofmt, vet, go test ./..., the Windows build and vet, and golangci-lint (0 issues) all pass on the merged branch.

The broker must keep a new batch or foreground lease waiting with a reason
that names the cordon, still grant a hold, and grant the waiter after an
uncordon. berth cordon and uncordon must work against a live daemon and
show in berth status. The gateway must treat the card as out of reach and
start no background run for it. These fail until the cordon lands.
berth cordon CARD stops new grants on a card, holds aside, while the
node's other cards serve; berth uncordon CARD lifts it and schedules the
card's waiters at once. A cordon drains nothing. Waiters say the card is
cordoned, berth status marks it CORDONED, and berth events records
card.cordoned and card.uncordoned. The gateway treats the card as out of
reach: no background engine starts there and a ComfyUI job for it queues.
The cordon lives in berthd's memory, so a restart clears it.

Third slice of v0.5, per-card gateway endpoints.
@T-Crypt
T-Crypt merged commit cdd41ca into next Oct 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant