Repository navigation
feat(gateway): place a plain request on the cheapest card - #280
Merged
Merged
Conversation
One model served on gpu0 and gpu1. A plain /v1 request must go to a parked copy before a free card, a free card before one a lease holds, and a held card before one a game pins. With nothing to tell the cards apart, the first entry wins. Three cases fail until the gateway ranks the cards.
When several entries serve the model on different cards and no card is named, resolve ranks each by what reaching it takes: a resume from RAM, a free card, a drain of the holder, then a card a VM or a pinned lease keeps. An engine that is up still answers first, and a tie keeps berth.toml's order. Second slice of v0.5, per-card gateway endpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of v0.5: per-card gateway endpoints.
What changes for users
When one model is served on several cards (one
[[engine]]per card, since #278) and a request uses the plain/v1URL, the gateway now picks the card. Before, it always took the first entry inberth.toml, so it could drain a busy gpu0 while gpu1 sat free, or queue behind a game.The order is:
cardFree): no lease, waiter, stowaway or active claim, and not in a VM.outOfReach), where the request waits.A tie keeps
berth.toml's order. A node with one entry per model behaves exactly as before.How
resolvecollects every entry that serves the model, then takesslices.MinFuncoverreach.reachranks an entry fromnode.Viewof its card. It reusesparkedIn,cardFreeandoutOfReach, the helpers the gateway already uses for parked routing, background engines and the ComfyUI queue.The roadmap's slice text now matches this order. It drops "the cheapest measured switch": the switch-cost book has no per-card data to rank on yet.
Proof
TestPlainRequestPicksTheCheapestCardhas five cases on a two-card berthd: both cards free, a game on gpu0, a lease on gpu0, a game on gpu0 with a lease on gpu1, and qwen parked on gpu1 through a real re-attach. Three of them fail on the test-only commit.TestCardBaseURLPicksTheEngineOnThatCardfrom feat(gateway): one base URL per card #278 still passes.go test ./..., the Windows build and vet, and golangci-lint (0 issues) all pass.