Repository navigation
feat(gateway): one base URL per card - #278
Merged
Merged
Conversation
One model served by an engine on gpu0 and another on gpu1. A request to /cards/gpu1/v1 must reach gpu1's engine, a card named by its PCI address must work, an unknown card is a 404 listing the cards, and the card's model list names only its engines' models. berth config check must accept the two entries. Both tests fail until the routes and the config rule land.
/cards/CARD/v1 takes chat, completions, embeddings, models and berth/wait. CARD is the card's name or PCI address. The gateway resolves the model among the engines whose entry runs on that card and strips the prefix before it proxies. berth config check now lets one model be served by one engine per card, and still refuses two on one card or where one names no card. A request without the prefix is routed as before. First slice of v0.5, per-card gateway endpoints.
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of v0.5: per-card gateway endpoints.
What changes for users
On a node with several cards, a client can now pick the card. Each card has a base URL on the gateway,
/cards/CARD/v1, whereCARDis the card's name (gpu1) or its PCI address. A request through it only reaches engines whose[[engine]]entry runs on that card, andGET /cards/CARD/v1/modelslists only their models. One model can be served on several cards, one entry per card on its own port, sohttp://127.0.0.1:9799/cards/gpu1/v1and.../cards/gpu0/v1reach different copies under the same model name.Nothing changes without the prefix. A plain
/v1request goes to the first entry that serves the model, as before.berth config checkstill refuses two entries that serve one model on the same card, or where either names no card.How
onCardwraps the chat, completions, embeddings, models andberth/waithandlers. It resolves the card through berthd's status with the same matchnode.Viewuses, puts the card's name in the path value, and strips the prefix so the engine sees/v1/....resolveandcatalogtake the card and keep only engines whose entry'sgpupicks it (node.CardsOf, exported fromcardsOf). An engine berthd finds with no entry inberth.tomlbelongs to no card, so only the plain URL reaches it.sharesCard).Proof
TestCardBaseURLPicksTheEngineOnThatCardruns two fake engines on a two-card berthd. It checks routing by card name and by PCI address, the unprefixed route, a model asked on the wrong card (404 naming the card), an unknown card (404 listing the cards) and the per-card model list. It fails before the feature commit.TestOneModelOnTwoCardsaccepts qwen on gpu0 and gpu1, and refuses gpu1 with gpu1, and no card with gpu1. It fails before the feature commit.resolvesends gpu1 requests to gpu0 and the test fails.go test ./..., the Windows build and vet, and golangci-lint (0 issues) all pass.Next slices
Placement among the copies for a plain
/v1request, then cordoning a card, then a card picker on the harbor board.