Raise the managed mesh peer-outage grace from 4 s to 300 s - #259
Open
sethforprivacy wants to merge 1 commit into
Open
Raise the managed mesh peer-outage grace from 4 s to 300 s#259sethforprivacy wants to merge 1 commit into
sethforprivacy wants to merge 1 commit into
Conversation
PeerWatch.transport_error latched mesh_failure after 4 s without an authenticated peer response. Peer health travels over the management LAN, not the ring fabric, so any management-switch event longer than 4 s (a reboot, firmware update or STP reconvergence) SIGKILLed a healthy model on all four ranks while the RoCE ring was untouched; with Restart=no the lane stayed down until an operator intervened. A 101 s switch reboot on 2026-09-09 did exactly that for 5 h. Generation changes, negative readiness, authentication failures and local marker exits keep their no-grace semantics; only the transport-unavailable path waits 300 s. MANAGED_MESH.md and the unit test are updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PeerWatch.transport_errorlatchesmesh_failureafter 4 s without anauthenticated peer response. Peer health is fetched over the management
LAN, not the ring fabric. Any management-switch event longer than 4 s (reboot,
firmware upgrade, STP reconvergence) therefore SIGKILLs a healthy model on all
four ranks while the RoCE ring is untouched, and with
Restart=nothe lanestays down until an operator intervenes.
Live evidence (2026-09-09, four-rank GLM-5.3-Flash MTP3 mesh)
enP7s7: link downon all eight hosts on the LAN in the same second (switch reboot);link up07:58:28Z (101 s).{"event": "mesh_failure", "error": "Authenticated peer transport remained unavailable beyond its grace interval"}(rank 3: "Management address does not identify this rank").docker kill -s 9of every model container; API had returned/health200 two seconds earlier. RoCE links ACTIVE, ring pings <1 ms throughout.up+start-model.Change
PEER_OUTAGE_GRACE = 300.0. Test and MANAGED_MESH.md updated. Generationchanges, negative readiness, authentication failures and local marker exits
keep their no-grace semantics, so the invariant "never authenticated negative
readiness or a new generation" is unchanged; only the tolerance for a silent
peer grows. A silent peer already stalls the collective, so the longer grace
defers cleanup, not correctness.
Alternative considered: make it a site/profile knob. Happy to do that instead if
preferred; 300 s as a default still seems right for a LAN-carried health path.