Repository navigation
Rollout check fails a deploy on one dropped public connection and cannot say why #5170
Description
🤖 Generated by the Agentic Engineer
Evidence
On 2026-10-20 23:39Z the production deploy of #5129 (a Kyverno policy change, unrelated to data products) failed at Verify the deployed data product UI release and removed #5129 from the merge queue:
{"complete":false,"failure":"public_contract_incomplete","phase":"public_contract","role":"descriptor_library","reason":"transport_failed"} (run 39513604314, job 118944678905)
- The rollout comparison itself passed. The failure was the last public fetch (
descriptor.js); earlier fetches to the same host in the same pass succeeded. - The file was healthy: 13,301 bytes, and three probes about two minutes later returned 200 in under 0.3 s.
- It is the only
transport_failedin the 215 deploys that reached this check since 2026-10-07 (read for Production deploys fail intermittently at the data product rollout check and evict unrelated pull requests #4545). Each eviction costs a re-queue and roughly one more deploy cycle (~30 min) for every PR behind it.
Problem
fetch() in scripts/verify-data-product-ui-rollout.sh maps only curl exits 6, 28, 35 and 60 to a named reason, and sends curl's stderr to /dev/null. Every other exit — a refused connection (7), an empty reply (52), a reset during receive (56) — becomes transport_failed. So the log cannot say what happened, and a single dropped connection fails the deploy outright even though the step has time left within its existing deadline.
Hypothesis
If the check names the curl exit code and retries a public fetch once when the connection itself failed, a one-off dropped connection stops evicting unrelated PRs, and any repeat names its cause.
Acceptance criteria
- A refusal from a public fetch carries curl's numeric exit code as a fixed field (no response data, hostnames or addresses in the log).
- A connection-level failure (exit 7, 52 or 56) is retried once within the remaining deadline; a retry never extends the deadline.
- DNS, TLS, timeout, HTTP status, body and header failures are not retried and keep their current reasons.
- The script's own tests cover: one dropped connection then success (passes), two dropped connections (fails with the exit code), and a non-retryable failure (fails on the first attempt).
Success signal
No deploy evicted by transport_failed over the next 4 weeks; any public-check refusal in that window names its curl exit.
Context: split from #4545, which closed after its own rollout-comparison criteria were met.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Oct 21, 2026
Second eviction from this check, a different reason code: on 2026-10-21 ~07:30Z the merge-group deploy of #5176 (a KSail version bump, unrelated to data products) failed with
{"complete":false,"failure":"public_contract_incomplete","phase":"public_contract","role":"kit_health","reason":"http_status"}(run 39559221660, job 119089422612)cluster updatechange summary and the credential reassert output are identical to the green fix(webapp): satisfy explicit container non-root admission #5162 deploy 35 minutes earlier (run 39555271352), so nothing the PR deploys differs at the point of failure./healthzafterwards returned 200 in under 0.3 s.Suggested scope addition: record the numeric HTTP status as a fixed field on
http_statusrefusals (same rule as the curl exit code: no response data or hostnames), and consider one bounded retry for a 502/503/504 from a public fetch within the existing deadline. 4xx and body/header failures would stay fail-first. Otherwise the acceptance criteria above ("HTTP status … not retried") will still let this signature evict unrelated PRs.devantler commentedon Oct 29, 2026
Rung-0 incident and WIP exception: the current merge-group deployment failed its public contract check at 15:22Z, then its automatic main recovery failed the same verifier at 15:47Z. The fixed roles differed (
release_index, thenpublisher_style); both refusals omitted the actual HTTP status. The gateway-specific checks passed. Subsequent bounded public responses do not explain those failures or verify complete recovery.I claimed this existing issue and started the diagnostic repair on 2026-10-29. Priority is Urgent because it blocks safe production delivery and recovery; the diagnostic slice is Low effort including local validation, CI, review and protected delivery. This is an explicit incident WIP exception while previously reviewed changes wait for verified recovery.
The first slice records only validated numeric HTTP/curl codes. Its behavior test failed against current main as expected. It adds no retries and preserves all body/header, rollout-stability and deadline gates. The issue remains open for cause analysis and its outstanding reliability acceptance; no four-week success or production recovery is claimed.
chore: agent-claim #5170 nonce=aa6975ada174ab73b948264e6407771b
devantler commentedon Oct 29, 2026
The diagnostic slice shipped in #5585 as
c5f9e66dc34950ecd80cd8889116f1568d3cfd48. Its protected candidate passed the production verifier with all 17 public checks complete and active readiness. Post-merge Validate Main, Release, Code Quality and CodeQL all succeeded. The native convergence job confirmed production runs that exact merged source, and v1.283.5 was published from it.#5170 remains open: the earlier intermittent connection/HTTP cause is unknown, the retry decision is outstanding, and the four-week reliability acceptance is not complete. This delivery adds numeric diagnostics while retaining the existing fail-first checks and deadlines.
chore: agent-claim #5170 nonce=23d362282f3f6a3a4283e8b07bcff83f
devantler commentedon Oct 29, 2026
Continuing the existing incident diagnostic work: both serving contract probes are Ready, but their diagnostic counters are absent from Coroot. The current pod template has no Coroot metric-discovery annotations, and its management port has no collector ingress allowance. The existing metrics identify completed failures, cancellation and contention without exposing response contents or credentials.
I am preparing a minimal discovery and exact-collector access repair, preserving the probe's target, image, checks and deadlines. This continues the recorded #5170 incident/WIP exception; the confirmed monitoring gap is the hotfix scope, not a claim that the historical public HTTP failure is diagnosed. Four bounded reads of each previously failing public route returned HTTP 200 today; those later observations do not establish its cause or long-term recovery.
chore: agent-claim #5170 nonce=94d1ed9a032207e8d32cdfbcb4a6393c