fix(distributed): exclude pinned models from cluster eviction and idle scale-down - #11872
fix(distributed): exclude pinned models from cluster eviction and idle scale-down#11872Dennisadira wants to merge 1 commit into
Conversation
…e scale-down pinned: true was only honoured by the per-node watchdog. Every distributed eviction path was pinned-blind: the router's LRU eviction (EvictLRU, evictLRUAndFreeNode) and the replica reconciler's idle scale-down would happily unload a pinned model — and since eviction is gated on in_flight = 0, a pinned model became eviction-eligible the instant each response completed. Under capacity pressure that surfaces as the backend being freed immediately after every request (mudler#11101). Wire the model config loader into the router and reconciler through a new PinnedModelResolver seam (mirroring ConcurrencyConflictResolver): - EvictLRU passes the pinned set into FindLRUModel's query so the next-oldest unpinned model is selected instead of the attempt failing - evictLRUAndFreeNode filters pinned models inside its locked selection - scaleDownIdle skips pinned models entirely: trimming to the floor still means requests beyond the survivor's capacity pay a cold reload Deliberate teardown (admin unload, model delete, node drain) intentionally still applies to pinned models, as does dead-row reaping (state correction, not eviction). Regression specs verified to fail with the exclusion disabled. Addresses the cluster-side eviction gap in mudler#11101 Assisted-by: Claude Code:claude-fable-5 [Claude Code] Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
|
This does not compile: the
Reproduced on the PR head with Fix is to pass The rest of the change looks good to me: the |
Description
Addresses the cluster-side eviction gap in #11101.
pinned: truewas only honoured by the per-nodeWatchDog(idle eviction, LRU limit, memory reclaimer, group eviction). Every distributed eviction path was pinned-blind:SmartRouter.EvictLRU→FindLRUModelSmartRouter.evictLRUAndFreeNode(locked SQL selection)ReplicaReconciler.scaleDownIdleSince eviction is gated on
in_flight = 0, a pinned model became eviction-eligible the instant each response completed. Under capacity pressure that surfaces exactly as the report describes: the backend is freed immediately after every request, with the reporter's settings (watchdogoff,max_active_backends: 0, reclaimer off) creating no watchdog at all, so the local exclusion never had a chance to apply.Fix
Wire the model config loader into the router and reconciler through a new
PinnedModelResolverseam (mirrorsConcurrencyConflictResolver):EvictLRUpasses the pinned set intoFindLRUModel's query, so the next-oldest unpinned model is selected instead of the attempt failingevictLRUAndFreeNodefilters pinned models inside its locked selectionscaleDownIdleskips pinned models entirelyDeliberate teardown (admin unload, model delete, node drain) intentionally still applies to pinned models, as does dead-row reaping.
Tests
core/services/nodes/pinned_eviction_test.goregression specs, verified to fail with the exclusion disabledGetPinnedModelNamesspecs in the config loader testsrevision_eligibility_test.goupdated for the newFindLRUModelsignatureDocs
docs/content/advanced/vram-management.mddocuments thatpinnednow also protects models in distributed mode.Notes for reviewers
The
pkg/grpcbuild on master currently needsmake protogen-gofirst (committed pb bindings lagbackend.proto); unrelated to this change.Fixes #11101