fix(bnk-vlans): retry kubectl apply on webhook-transient errors - #65
Merged
Merged
Conversation
Endpoint registration fires on kubelet's readiness probe, which on a cold-start cne-controller pod precedes the in-container TLS server binding port 3340 by ~2-3 minutes. The previous 150s endpoint wait + single apply attempt routinely failed against "connection refused" during that gap. Replace the single attempt with a bounded retry loop that only retries on webhook-transient errors (connection refused, no endpoints available, failed to call webhook, i/o timeout, context deadline exceeded) and fails fast on validation rejections or kubeconfig errors so real bugs aren't masked. Budget: 300s @ 15s intervals. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bonnyr-f5
force-pushed
the
fix/vlans-webhook-wait
branch
from
May 19, 2026 13:24
7919cf0 to
2b36e07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the fixed 3x-sleep retry (introduced in #54) with a bounded retry loop (300s budget, 15s interval) that distinguishes webhook-transient errors from real failures.
connection refused,no endpoints available,failed to call webhook,i/o timeout,context deadline exceeded.Motivation
During a dual_dpu_obmc deploy on mgx-19 (2026-05-19), the 3-attempt approach from #54 still wasn't enough on a cold-start
f5-cne-controllerpod. The endpoint registers (kubelet readiness probe passes) before the in-container TLS server binds port 3340, and the 15s + 30s + 30s schedule lands inside that gap. Manualkubectl applyworked after another ~60s; the deploy succeeded on UI retry.The fixed retry schedule has two failure modes:
The bounded retry with error-class matching addresses both.
Test plan
f5-cne-controllerpod cold-starts; verify the loop reports "Webhook transient at Xs:" log lines and then succeedsNote for reviewer
This branch is currently behind
release/2.2(merge-base predates #52 and #54). Net diff against currentrelease/2.2is the single bounded-retry replacement inbnk/bnk-vlans/main.tf. Rebase before merging if you want a linear history.🤖 Generated with Claude Code