EtcdCluster status freezes at its last good value when etcd is unreachable

[lllamnyp]

Summary

When the operator cannot reach an EtcdCluster's etcd endpoints, the cluster reconciler returns early and never reaches updateStatus. status.readyMembers, Available, Degraded and Progressing keep whatever values they had the last time etcd answered. A cluster whose every member is down keeps reporting Available=True / QuorumHealthy / "All members are ready" and readyMembers: 3 indefinitely, while its EtcdMembers all report Ready=False.

Observed on a Cozystack integration cluster running etcd-operator v0.5.4 after a graceful node shutdown took every etcd pod down at once. main (a3d1b19) has the same code.

What it looks like

$ kubectl get etcdclusters -A
NAMESPACE         NAME   VERSION   REPLICAS   READY   AGE
tenant-portal     etcd   3.6.11    3          3       5d7h

$ kubectl get etcdmembers -A
NAMESPACE         NAME         CLUSTER   VERSION   RUNNING   READY   AGE
tenant-portal     etcd-7p9fr   etcd      3.6.11    3.6.11    False   5d7h
tenant-portal     etcd-9wm4c   etcd      3.6.11    3.6.11    False   5d7h
tenant-portal     etcd-smx9c   etcd      3.6.11    3.6.11    False   5d7h

EtcdCluster conditions (lastTransitionTime a day older than the outage):

Available:   True   QuorumHealthy  "All members are ready"
Degraded:    False  QuorumHealthy
Progressing: False  Reconciled     "actual state matches status.observed"
readyMembers: 3

Every EtcdMember:

Ready: False  PodNotReady  "pod phase: Succeeded"

Operator log, one entry per cluster every ~15 s, nothing else:

ERROR  MemberList failed  {"controller": "etcdcluster", ..., "error": "context deadline exceeded"}
  ... dial tcp: lookup etcd-7p9fr.etcd.tenant-portal.svc on 10.96.0.10:53: no such host
github.com/cozystack/etcd-operator/controllers.(*EtcdClusterReconciler).promotePendingLearner
	controllers/etcdcluster_controller.go:1010
github.com/cozystack/etcd-operator/controllers.(*EtcdClusterReconciler).Reconcile
	controllers/etcdcluster_controller.go:344

Code path (main @ a3d1b19)

Reconcile, steady-state branch (current == desired):

• controllers/etcdcluster_controller.go:344 calls promotePendingLearner.
• controllers/etcdcluster_controller.go:1008-1011: MemberList fails, the error is logged, and the function returns &ctrl.Result{RequeueAfter: 10s}, nil.
• controllers/etcdcluster_controller.go:348-350: if res != nil { return *res, nil } returns from Reconcile.
• controllers/etcdcluster_controller.go:379: updateStatus is never reached.

updateStatus (controllers/etcdcluster_controller.go:1334) is the only place that recomputes readyMembers from the members' Ready conditions and derives Available / Degraded from that count. Had it run it would have written 0/3 members ready, quorum lost. The PDB reconcile also lives there, so the PDB goes stale as well.

The sibling error branches directly above (TLS config, credentials, dial failure, lines 328-341) each carry a comment saying not to bail out of the whole reconcile because updateStatus still has useful work. The MemberList-failure branch does exactly what those comments warn against.

Why it matters

Anything that gates on Available (Flux/Argo health checks, alerting, the tenant Kubernetes control plane in Cozystack, a human running kubectl get etcdclusters) is told the cluster is healthy precisely when it is not. The failure mode is silent: the operator is not crashing, the object is being reconciled every 15 s, and the status looks fresh.

Things to think about before fixing

The one-line fix (return nil, nil from the MemberList-failure branch so the reconcile falls through to updateStatus) addresses this instance. Some broader questions are worth settling at the same time rather than after the next occurrence:

1. Should status ever depend on etcd being reachable? updateStatus only needs the EtcdMember list from the apiserver. Structurally, status derivation could run unconditionally (for example via defer, or by moving it ahead of every etcd-dialling step) so that no future early return can freeze it again. Reconcile has around a dozen return &ctrl.Result{RequeueAfter: 10s} sites in scaleUp / completePendingMember / tryPromoteLearner (lines 1144-1198) that would all skip status today.
2. Is "all member pods Ready" the right definition of Available? A member's Ready mirrors its Pod's readiness condition. A pod can be Ready while etcd has lost quorum (partition, disk stall) and the operator would still report QuorumHealthy. A MemberList/Status RPC failing across every endpoint is itself strong evidence that quorum is gone; it may deserve to feed Available directly (there is already an Available=False / ClusterUnreachable reason used during discovery at line 684).
3. Should status carry a staleness signal? Something like an observed timestamp or an Unknown condition status when the operator has not been able to verify the cluster for N reconciles, so consumers can distinguish "verified healthy" from "last known healthy".
4. Test coverage. An envtest/unit case where MemberList fails while members are not Ready, asserting Available flips to False, would have caught this and guards the fix.

Related: the members in this incident stayed down because their Pods, having reached a terminal phase, are never replaced (#368).