Defer local management-address loss to the peer grace bound - #266
Draft
FujitsuPolycom wants to merge 4 commits into
Draft
Defer local management-address loss to the peer grace bound#266FujitsuPolycom wants to merge 4 commits into
FujitsuPolycom wants to merge 4 commits into
Conversation
A missing management address raised ValueError out of the first leg of _links() and killed the rank within one poll, before any fabric check ran; the Sep 9 switch-reboot incident killed all four ranks through this path while the RoCE ring stayed healthy. Split the management probe from the fabric checks: _links() now records the absent address and still verifies MAC, MTU, GID, netdev and RDMA MTU, and check() raises the new typed ManagementAddressLoss only after every fabric and object check passed. The run loop catches that type around its network check, latches a PeerWatch.management_error timer with the same PEER_OUTAGE_GRACE bound, publishes management_degraded for admission gating, and keeps serving; recovery clears the latch only when its own condition recovers. Peer transport and management latches are independent. Startup (up(), bind) still fails on the first absent address, and any fabric fault, marker exit, authentication failure, generation change, negative readiness, model exit or management-identity mismatch still fails the rank immediately. validate_group refuses a management_degraded rank. Validation: 47 managed-service tests (six new PeerWatch latch/admission cases and two loop-driven cases covering in-grace survival and expiry and the fabric-fault precedence), 616 mesh-suite tests, Ruff clean.
Add the startup/runtime split to the readiness table and state the grace-bound semantics: fabric, marker, authentication, generation and readiness checks keep their immediate-failure semantics during the interval, a differing management identity never receives grace, and the model-process survival does not imply management-network availability for API clients.
Address review: grace now requires that the exact rank/address/netdev tuple passed startup validation. up() records validated_management_identity after its _links() and object checks succeed; a runtime check raises the typed ManagementAddressLoss only when the recorded tuple still matches the site's current assignment, and raises ValueError otherwise — before the first successful up() the address absence never receives grace. The in-process site-edit mismatch case is impossible (the site document is loaded once at construction), so the only unvalidated path is the pre-startup one, and the docstring now states that precisely. Tests are now parameterized against PEER_OUTAGE_GRACE instead of hardcoded 4-second timings, so they hold when #259 raises the bound to 300 s: the latch test steps 0.1 s inside the bound and asserts at exactly the bound, and the run-loop expiry phase advances two grace periods per round. The fabric-fault test no longer asserts an empty loss record — the recorded loss is irrelevant once the fabric ValueError takes precedence — and a new startup test pins that up() with the address absent fails before any validation is recorded. Real-NetworkManager coverage (test_managed_network.py, mocked command results, no fabricated supervisor stubs): typed loss raised only after every port/GID/netdev/object check ran with the address absent; the RDMA MTU fault on the same round wins over the recorded loss; a missing TC rule wins; the pre-validation absence fails fast through the loss channel; and startup with the address absent records nothing. Validation: 35 managed-network tests, 47 managed-service tests, 621 mesh-suite tests, Ruff clean.
up() now sets validated_management_identity after its links check, the conflict-detection pass and the object creation/verification loop all succeeded, matching the description: the grace bound covers exactly the configuration the supervisor verified at startup, not a partially validated one. The all-fabric-checks-ran assertion now slices host.commands to the commands issued during the outage itself, so startup activity cannot make the test pass falsely. Verified by mutation: skipping the links call inside check() makes the test fail. Executed the grace-sensitive tests under both PEER_OUTAGE_GRACE values: the four PeerWatch latch/admission tests and both run-loop tests pass at 4.0 and at 300.0 (the post-#259 value), not just by arithmetic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Layered on #259: a temporary loss of this rank's own management address now
shares the peer-transport grace bound (300 seconds) instead of killing the
rank within one poll. Tracks the follow-up identified when #259 landed:
#259 defers only peer silence; the local leg still failed immediately,
and the 2026-09-09 management-switch reboot killed rank 3 through exactly
that path (
Management address does not identify this rank) while theRoCE ring stayed healthy.
Change
managed_network.py: the management-address probe is split out of thefabric checks. With a loss channel passed, an absent address is recorded
and every fabric check still runs — MAC, MTU, port IP, GID/netdev
identity, RDMA active-MTU probe and object-presence checks — so a RoCE
fault is never delayed by a management outage.
check()raises the newtyped
ManagementAddressLossonly when every fabric check passed. Thetyped loss is granted only to the exact rank/address/netdev tuple that
passed startup validation:
up()recordsvalidated_management_identityafter all startup checks succeed, and anabsent address before that point (or on any other unvalidated path)
raises
ValueErrorand never receives grace.up()and the healthserver bind keep their fail-fast startup semantics.
managed_service.py: the run loop catchesManagementAddressLossaround its network check, latches an independent
PeerWatch.management_errortimer sharingPEER_OUTAGE_GRACE, andpublishes
management_degradedwhile serving continues. The latchesare independent: a successful peer poll clears only the peer latch, and
management recovery clears only the management latch. Model-exit
detection, marker watchdog, and the systemd watchdog keep running every
round during the outage.
validate_grouprefusesmanagement_degradedranks for new model admission.(which now always execute during a management outage and take
precedence over the recorded loss), marker exits, authentication
failures either direction, negative readiness, generation changes, and
model-exit evidence.
MANAGED_MESH.md: readiness-table row for the management-address probeand a paragraph stating the semantics, including the availability
caveat: the model process survives the outage, but API clients routed
over the management network can still lose connectivity for its
duration, and a peer fault visible only through the management path is
detected up to the grace interval later than before.
Validation
test_managed_network.pytests, including five newreal-
NetworkManagercases with mocked command results: the typed lossis raised only after every port/GID/netdev/object check ran with the
address absent; an RDMA-MTU fault or a missing TC rule on the same
round takes precedence; a pre-validation absence fails fast through the
loss channel; and
up()with the address already absent fails beforerecording any identity.
test_managed_service.pytests: latch expiry at exactlyPEER_OUTAGE_GRACE, own-condition recovery, independent timers in bothdirections, admission blocking, and two loop-driven cases — in-grace
survival (fabric check on its 5-second cadence, peer checks and watchdog
ticks every round, degraded flag set, clean exit) and sustained expiry
into fail-closed — plus a combined-fault case proving a fabric fault
fails the rank immediately while the management latch is armed.
PEER_OUTAGE_GRACEand were executedat both 4.0 (pre-Raise the managed mesh peer-outage grace from 4 s to 300 s #259 value) and 300.0; the rebased tree runs with
300.0 and the full mesh suite passes (621 tests, 14 skips). Ruff, 766
repository-relative Markdown links and release-safety checks pass.
the outage (startup commands excluded), and was mutation-verified:
skipping the fabric walk inside
check()fails the test.Untested
Hardware outage survival is not tested: the grace behavior under a real
management-switch reboot, gateway failover, or RoCE-fabric flap has not
been exercised on the four-rank mesh. The controlled fault drill (drop
the management address on one rank while serving and observe the
degraded-but-serving state, then expiry) remains open; the timeline in
MANAGED_MESH.mddescribes the designed behavior, not a measured one.Whether a gateway failover also drops routes/qdisc objects on the
management netdev (which would make the object checks fail immediately
during such an outage) is unknown until that drill runs.
Merge notes
Rebased onto
84f2a01(#259). One documentation conflict resolved infavor of #259's 300-second wording; the management-address table row
references the peer-transport bound without restating the number.