Repository navigation
Prevent repeated database preemption by essential controllers #4748
Description
🤖 Generated by the Agentic Engineer
Description
A database standby was preempted three times during the current rollout, reducing redundancy while each replacement started. Read-only scheduler events identify preemption, and the most recent preemptor's identity joins to a system-critical Flux controller. Database-operator logs independently record the unhealthy interval, replacement creation, and recovery. All database instances were Ready again in the subsequent read; no primary change or data-loss evidence was observed.
Steps to reproduce
- Exercise controller scheduling during a protected rollout with storage-backed database instances present.
- Join each database preemption event to the preempting Pod's immutable identity and priority.
- Observe the database's redundancy and replica recovery, rather than relying only on the final Ready count.
Expected behavior
Essential controllers can schedule without repeatedly sacrificing database redundancy. Preserve system-critical ordering, disruption budgets, real outage reporting, and storage replica restrictions.
Actual behavior and scope
The database uses ordinary priority. PR #4650 addresses preemption by the storage scheduling tier, but deliberately keeps databases below system-critical controllers; it does not fix this newly observed case. Existing client-placement work #4718 may relieve the scheduling constraint, but that effect is not yet proven. Coroot's separate instance-binding issue #4720 can extend a displayed warning after recovery and is not a substitute for fixing the real disruption.
Acceptance criteria
- Establish which capacity or placement constraint makes controller scheduling repeatedly choose a database victim, using complete observations and identity joins.
- Reproduce the selected correction against the real scheduling behavior, with a failing baseline and passing repair; do not count manifest validation alone.
- Preserve controller availability and critical priority, database data and primary identity, disruption budgets, and storage safety. Do not elevate databases above essential controllers or hide the alerts.
- Deliver the smallest declarative repair through current-head review, required CI, and protected deployment.
- Verify separated rollout samples maintain database redundancy and that the original preemption pattern does not recur. If Restore volume attachment on compute-only workers #4718 demonstrably resolves it, record that evidence instead of adding duplicate configuration.
Activity
devantler commentedon Oct 11, 2026
The three standby preemption events occurred during repeated security-handoff restarts of the Flux kustomize controller. The three generated controller templates are identical after removing the restart timestamp and ReplicaSet hash. The restart is intentional: the handoff must retire every process that could retain the preceding credential. It is not safe to suppress that boundary.
The controller uses two required cross-node replicas with a stable-readiness floor. The default rolling strategy rounds to one surge replica and zero unavailable replicas. The failed scheduling events report both insufficient memory and required anti-affinity exclusions; the latest preemptor joins by immutable Pod identity to this system-critical controller. The operator independently reports the standby's unhealthy interval, replacement and recovery. The primary identity remained unchanged in the separated reads.
Current OIDC reads again show all four database clusters fully Ready and configured archiving healthy, with every node Ready and uncordoned. The existing capacity buffer is active, but its template reserves CPU/memory without the controller's required anti-affinity: a generic spare-capacity observation is not proof that a third controller replica fits. This is a correction candidate to reproduce, not yet a tested repair.
Preserve the security restart, critical ordering, readiness floor, cross-node placement, and data protections. First verify the reviewed storage-client placement repair's effect (#4718 / #4744); if the preemption pattern remains, reproduce affinity-aware rollout headroom against the actual scheduler before changing configuration. Do not count the current recovered Ready state as proof that the disruption is resolved.