Repository navigation
feat: berth cordon and uncordon a card - #281
Merged
Merged
Conversation
The broker must keep a new batch or foreground lease waiting with a reason that names the cordon, still grant a hold, and grant the waiter after an uncordon. berth cordon and uncordon must work against a live daemon and show in berth status. The gateway must treat the card as out of reach and start no background run for it. These fail until the cordon lands.
berth cordon CARD stops new grants on a card, holds aside, while the node's other cards serve; berth uncordon CARD lifts it and schedules the card's waiters at once. A cordon drains nothing. Waiters say the card is cordoned, berth status marks it CORDONED, and berth events records card.cordoned and card.uncordoned. The gateway treats the card as out of reach: no background engine starts there and a ComfyUI job for it queues. The cordon lives in berthd's memory, so a restart clears it. Third slice of v0.5, per-card gateway endpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third slice of v0.5: per-card gateway endpoints.
What changes for users
On a node with several cards,
berth cordon gpu1closes a card to new work while the other cards keep serving: for maintenance, a driver test, or before a VM takes the card.berth uncordon gpu1reopens it. The card can be named by its name or its PCI address.berth stop ENGINEdoes that.berth hold, a game claim) is still granted.berth statusmarks the cardCORDONED, andberth eventsrecordscard.cordonedandcard.uncordoned./v1request goes to another card that serves its model (feat(gateway): place a plain request on the cheapest card #280's ranking), no background engine starts for the card, and a ComfyUI job for it queues.How
cordonedflag.evaluatereturns early for any waiter belowHold,Cordon(on)emits the event and schedules at once, andStatussetsGPU.Cordoned.OpCordonandOpUncordonwithCard, answered with the cards' names inCards.node.CordonusesPick, so a comma list or a PCI address works as it does forberth run --gpu.berth cordonandberth uncordon, modelled onberth stop, with JSON and YAML output.cardFreeandoutOfReachcheck a cordoned card.reach(from feat(gateway): place a plain request on the cheapest card #280) builds on both, so a cordoned card ranks last.The branch merges
nextafter #280. That merge commit also resolves the roadmap conflict, adds a "gpu0 cordoned" case to the #280 placement test, and adds the cordon to the gateway page's placement list.Proof
TestCordonStopsNewGrantsUntilUncordoned(broker): a foreground waiter is kept with a reason naming the cordon, a hold is granted, and the waiter is granted after the uncordon. Status shows the flag both ways.TestCordonAgainstADaemon(CLI): cordon and uncordon through the socket,CORDONEDinberth status, the JSON flag, and exit 1 for an unknown card.TestCordonedCardIsNotFree(gateway):outOfReachnames the cordon on gpu1 and nothing on gpu0, and no background run is queued for gpu1 until the uncordon. With the two gateway changes reverted, both assertions fail.go test ./..., the Windows build and vet, and golangci-lint (0 issues) all pass on the merged branch.