Repository navigation
Talos node upgrades can exceed the 10-minute drain timeout, evicting the bump from the merge queue #3388
Description
🤖 Generated by the Agentic Engineer
Evidence
A Talos patch bump (v1.13.8 → v1.13.9, #3382) was evicted from the merge queue when the prod deploy's 🔄 Update cluster (Talos machine config) step failed after ~16 minutes:
✗ distribution upgrade to pinned version v1.13.9 failed: rolling upgrade from v1.13.8 to v1.13.9:
upgrading node <redacted> (worker): drain: drain node prod-worker-2 (timeout 10m0s):
error when evicting pods/"instance-manager-…" -n "longhorn-system":
global timeout reached: 10m0s; raise spec.cluster.talos.drainTimeout (or --drain-timeout)
to give workloads more time, or re-run with --force-drain to delete pods bypassing PodDisruptionBudgets
The drain retried the same eviction in a tight loop for the full window, each attempt refused with Cannot evict pod as it would violate the pod's disruption budget.
The mechanism is a timeout mismatch between two correct settings, not a flake:
- Longhorn runs
node-drain-policy: block-if-contains-last-replica(confirmed live). It deliberately refuses to drain a node holding a volume's last healthy replica until it has rebuilt one elsewhere. - The live
instance-manager-*PDBs each reportALLOWED DISRUPTIONS: 0, so this is structural, not transient. ksail.prod.yamlsets nospec.cluster.talos.drainTimeout, so ksail's 10-minute default applies — less than a replica rebuild needs.
Failing run: 32956840398 (2026-08-26 10:10Z).
Impact
Every Talos version bump risks evicting itself — and whatever else is queued behind it — from the merge queue. Each occurrence costs a full ~35-minute prod deploy that drains and uncordons real nodes before failing. #3382 needed three queue attempts to land.
Because the drain is retried in a loop and the deploy job's later steps (Cilium gate, GHCR-auth reassert) run as always-steps and succeed, the job log tail reads green — the failure is only visible at the cluster_update end-action. That makes this easy to misread as an unrelated deploy fault.
Expected
A Talos node upgrade either drains within its allotted window, or waits long enough for Longhorn to hand off the last replica — so a routine patch bump merges in one queue attempt.
Acceptance criteria
- Measure how long Longhorn actually needs to relocate the last replica off a drained node, at current volume sizes.
- Set
spec.cluster.talos.drainTimeoutinksail.prod.yamlto comfortably exceed that, and record why that value. - Confirm the deploy still fails promptly on a genuinely stuck drain rather than hanging for the new timeout.
- Prove it: a Talos patch bump merges in a single queue attempt.
Explicitly NOT the fix
--force-drain deletes pods bypassing PodDisruptionBudgets. Here the PDB is the data-safety control — it is protecting a volume's last healthy replica — so forcing the drain trades a merge-queue annoyance for a risk of data loss on the storage layer. Likewise, relaxing node-drain-policy away from block-if-contains-last-replica removes the protection rather than accommodating it.
Size
Small — one config field plus the measurement that justifies its value.
Activity
devantler commentedon Aug 26, 2026
Refinement: the "last replica" is transient, and the rolling upgrade creates it
Live steady state (read-only,
admin@prod) shows the block is not caused by any under-replicated volume:numberOfReplicas: 3,state=attached,robustness=healthy.prod-worker-1,-2and-3each hold 13 running replicas, one per volume.So draining a single worker in steady state leaves two healthy replicas and
block-if-contains-last-replicawould not fire.It fires because a rolling upgrade reboots the workers in sequence. While a previously-upgraded worker's 13 replicas are still rebuilding, the node being drained next genuinely does hold the last healthy replica for some volume — so Longhorn correctly blocks, and keeps the per-node
instance-managerPDB atALLOWED DISRUPTIONS: 0until the rebuild catches up.The upgrade is therefore outpacing Longhorn's replica rebuild, and 10 minutes is not enough slack for ~13 replicas to re-sync between nodes.
This sharpens acceptance criterion 1: the quantity to measure is replica re-sync time between consecutive node reboots during a rolling upgrade, not a static per-volume rebuild time. It cannot be measured from steady state — every volume is healthy right now — so it needs capturing during the next Talos bump.
It also reinforces the "explicitly not the fix" note:
--force-drainwould delete the instance-manager holding the only healthy replica of a volume mid-rebuild, which is precisely the case the policy exists to prevent.devantler commentedon Oct 2, 2026
Progress check, 2026-10-02: AC1 measured from logs, AC2 already shipped, AC3–AC4 still open
AC1: measured from three rolling upgrades' deploy logs. I timed the gap between each worker's
Draining …line and itsUpgrading … via LifecycleServiceline:drainTimeout: 40m)What this shows:
block-if-contains-last-replicawait does not happen on every rollout.AC2: met by #3916 (merged 2026-09-20).
ksail.prod.yamlsetsspec.cluster.talos.drainTimeout: 40m, with the reasoning recorded next to it: CloudNativePG's 1800 s stop window plus 10 minutes for Longhorn to release the instance-manager PDB, after a 30m budget failed at exactly that deadline. The #4099 roll above ran under it and did not come near it.AC3: open. No run has measured how long a deploy takes to fail when a drain is genuinely stuck. A 40m per-node budget makes that interaction with the queue relevant, because merge-group deploys already compete with the queue's check timeout (#3097).
AC4: open, waiting for a vehicle. No Talos patch bump has gone through the queue since #3382. Talos v1.13.11 was published on 2026-10-01, but Renovate's
renovate/talosbranch is occupied by the v1.14.0 PR (#4044), which is parked on #3850. A v1.13.11 patch bump would therefore be the natural single-attempt proof.Next: when a Talos patch bump merges, record its queue-attempt count and per-worker drain times here. If it merges in one attempt with all drains under 40m, AC2 and AC4 are met, and AC3 is the only thing keeping this open.
chore: agent-claim #3388 nonce=8beb00f98a620681ba4ef585c1281272
devantler commentedon Oct 6, 2026
AC4 now has a vehicle on the way: #4557.
Production runs Talos v1.13.10 and v1.13.11 has been published since 2026-10-01, but no update was proposed. Renovate offers only the newest non-major release, which is v1.14.x, and that pull request (#4044) is parked on its migration. So the wait recorded above was not for upstream to publish a patch: the patch existed and could not be offered.
#4557 gives Talos patch releases their own pull request beside the parked minor one, still never auto-merged and with the same manual checklist. Once it merges, the v1.13.11 pull request Renovate opens is the single-attempt proof this issue is waiting for: record its queue-attempt count and per-worker drain times here.
devantler commentedon Oct 6, 2026
#4557 merged at 09:15Z: Renovate now offers Talos patch releases on their own branch (
renovate/patch-talos) while the v1.14 minor upgrade stays parked in #4044. Production runs v1.13.10 and v1.13.11 has been out since 1 October.Next step: when Renovate opens the patch pull request, complete its checklist (talosctl checksum, ISO, expected version), drive it through the queue, and record the node drain times here.
devantler commentedon Oct 7, 2026
The Talos patch pull request is not late: it is due after 2026-10-08 17:54Z. No
renovate/patch-talosbranch exists yet, 25 hours after #4557 merged, and that is expected. This repository makes Renovate wait seven days after a release before offering it, and Talos v1.13.11 was published on 2026-10-01 at 17:54Z. Renovate itself is running normally (it opened and merged #4619 today).Next check: after that time. If the patch pull request has still not appeared by 2026-10-09, the split added in #4557 is not doing what it was meant to and needs another look; if it has, the steps in the previous comment apply.
devantler commentedon Oct 10, 2026
The protected deployment failure reproduced with the configured 40-minute drain window in run 38091510339. The failing step is autoscaler boot-image reconciliation: pod eviction remains blocked by database-primary and storage disruption budgets until the global drain timeout. Later successful admission and gateway checks do not make that failed deployment green.
A read-only follow-up found the declared nodes Ready and schedulable and the affected databases fully Ready. That establishes current readiness only; it does not establish the requested boot image, successful deployment, or alert clearance. The protected recovery job is still restoring main. No budget was disabled and no pod, node, volume, or database was changed by this investigation.
The next diagnosis must distinguish the database operator's supported switchover from storage replica evacuation and verify each against the failed drain interval. Increasing the timeout again is not sufficient evidence of a repair. Preserve the disruption budgets and data-protection rules; do not re-queue the failed rollout until recovery and the drain prerequisite are verified.
The bounded storage-manager log confirms it retained the disruption budget because volume engines were still attached. That message does not prove a last-replica rebuild failure. The available database-operator log begins after the failed drain, so it cannot establish why the expected switchover did not complete during that interval. Preserve that uncertainty before choosing a configuration change.
Recovery readback (2026-10-11 00:32 UTC): the protected restore-main job completed successfully at 00:18:17 UTC. Its reviewed production checks passed, and a fresh consumer platform-health check exited 0. This clears the recovery-pending state; the original 40-minute drain failure remains unresolved and is still the investigation scope of this issue.