Skip to content

fix(vpa): keep cpu and memory requests in the CSI and module controller pods - #230

Merged
duckhawk merged 1 commit into
mainfrom
fix/vpa-requests-in-pod-templates
Oct 28, 2026
Merged

duckhawk merged 1 commit into
mainfrom
fix/vpa-requests-in-pod-templates

Conversation

@duckhawk

@duckhawk duckhawk commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Description

With vertical-pod-autoscaler-crd enabled, helm_lib_csi_controller_manifests, helm_lib_csi_node_manifests and helm_lib_module_controller_manifests stop rendering the cpu and memory requests of their containers and leave them to the VPA admission controller. They now always render them: the same values the VerticalPodAutoscaler uses as minAllowed (provisioner_resources, node_resources, helm_lib_module_controller_resources and so on).

Unit tests cover all three templates with vertical-pod-autoscaler-crd enabled. They fail on main and pass with the change. The chart version goes to 1.72.34.

No restart of critical cluster components. Pods of the affected workloads are recreated once, when a module picks up the new lib-helm, because the pod template changes.

Why do we need it, and what problem does it solve?

Until the admission controller sets requests on a new pod, the pod has an ephemeral-storage request only and its QoS class is BestEffort. The VPA updater in InPlaceOrRecreate mode cannot resize a BestEffort pod in place:

"In-place resize failed, falling back to eviction" error="Pod \"csi-controller-f548dd8fc-9bfhs\" is invalid: spec: Invalid value: \"BestEffort\": Pod QOS Class may not change as a result of resizing"

It evicts the pod instead. The replacement is BestEffort as well, and the loop goes on until the admission controller starts setting requests.

On a fresh e2e cluster of the csi-mind-ustor module, the CSI controller was evicted every 30 seconds for about seven minutes after its VerticalPodAutoscaler appeared (12:56–13:02, one eviction per updater loop). The CSI sidecars hold their leader leases for 30 seconds, so none of them ever became leader, and a PersistentVolumeClaim stayed Pending for the whole loop. The csi-node and module controller pods went through the same evictions. On main of that module the same step took 95 seconds instead of the usual 15.

_cloud_controller_manager.tpl, _cloud_data_discoverer.tpl and _capi_controller_manager.tpl follow the same pattern and have tests that pin it. They are left as they are so their owners can decide.

What is the expected result?

  • The containers of the CSI controller, the CSI node plugin and the module controller (with webhooks and kube-rbac-proxy) have cpu and memory requests in their pod templates whether or not vertical-pod-autoscaler-crd is enabled.
  • A new pod of these workloads is Burstable from the start. The VPA updater resizes it in place and does not evict it in a loop; no EvictedByVPA events right after a module is enabled.
  • The admission controller still replaces the requests with the recommendation once it has one.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

…er pods

With vertical-pod-autoscaler-crd enabled, the CSI controller, CSI node and
module controller templates left the cpu and memory requests out and relied on
the VPA admission controller to set them when a pod is created. Until it does,
the pod has ephemeral-storage requests only and is BestEffort. The VPA updater
in InPlaceOrRecreate mode cannot resize such a pod in place ("Pod QOS Class may
not change as a result of resizing") and evicts it instead; the replacement is
BestEffort again, and the loop goes on until the admission controller starts
setting requests.

On a fresh e2e cluster of csi-mind-ustor the CSI controller was evicted every
30 seconds for about seven minutes after its VerticalPodAutoscaler appeared.
The sidecars hold their leader leases for 30 seconds, so none of them became
leader and a claim stayed Pending for the whole loop.

The templates now always render the requests the VerticalPodAutoscaler uses
as minAllowed. The pod is Burstable from the start, the admission controller
still replaces the requests with the recommendation when it has one, and the
updater resizes the pod in place instead of evicting it.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@duckhawk

Copy link
Copy Markdown
Member Author

Checked on the e2e suite of the csi-mind-ustor module, on fresh clusters with VPA enabled:

lib-helm First volume lifecycle spec Suite
1.72.32 (four runs) PVC Pending for 5 minutes, CSI controller evicted every 30 s by the VPA updater failed
this branch (9a0b711), packaged as 1.72.34 15.2 s 31/31 passed

The module had no other change between the last failing run and the passing one.

@kkozoriz
kkozoriz self-requested a review October 28, 2026 16:36
@duckhawk
duckhawk merged commit 12ab8a1 into main Oct 28, 2026
4 checks passed
@duckhawk
duckhawk deleted the fix/vpa-requests-in-pod-templates branch October 28, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants