Repository navigation
feat: trim the CSI and module controller memory footprint - #234
Merged
Merged
Conversation
The CSI controller is one of the largest consumers of memory requests on master nodes: every replica runs the driver and five sidecars, and each of those processes gets its own memory request from the VPA. Measured on DKP 1.74.18 and 1.76.11 with nine storage modules, the sidecars and drivers on the three masters peak at 5.3 GiB of working set while their requests reach 9.5 GiB (1.72-1.75), 18.4 GiB (1.76.0-1.76.11, 64Mi rounding) and 11.0 GiB (1.76.12 and later). - In HA mode the controller now runs two replicas instead of one per master. Every sidecar is leader-elected, so the third replica only adds another standby. A cluster with one master keeps one replica, and the rollout keeps one replica running (maxSurge 0, maxUnavailable 1). - attacherEnabled (default true) lets a driver whose CSIDriver has attachRequired: false drop the external-attacher container and its VPA policy. Kubernetes creates no VolumeAttachment for such a driver, so the attacher has nothing to do. - resizerHandleVolumeInUseError (default true) passes --handle-volume-inuse-error=false to the external-resizer. With the upstream default the resizer starts an informer over every Pod in the cluster to wait for in-use volumes to be released; a driver that expands volumes online never reports them, and the leader resizer then holds a cache that grows with the number of pods (about 60 MiB against 16 MiB on a standby). The flag exists in every resizer version we ship (v1.8.0 and later). Both new options keep the current rendering when unset. Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
helm_lib_module_controller_manifests with onMasterNode ran one replica per master in HA mode, like the CSI controller did. The controller is leader-elected, so with three masters the third replica only adds a standby, along with its webhooks and kube-rbac-proxy containers and their memory requests. Move the two-replica rule from the CSI controller into helm_lib_module_controller_on_master_strategy_and_replicas_for_ha and use it for both: two replicas in HA mode, one on a cluster with a single master, maxSurge 0 and maxUnavailable 1. Without HA the rendering is unchanged. A module controller off masters already ran two replicas. Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
3 of 4 tasks
kkozoriz
self-requested a review
November 15, 2026 10:49
kkozoriz
approved these changes
Nov 15, 2026
2 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
helm_lib_csi_controller_manifestsandhelm_lib_module_controller_manifestsget changes that cut the number of processes, and with them the memory requests, of the CSI and module controllers on master nodes:csiControllerHaMode: trueand the module controller withonMasterNode: true(the default) ran one replica per master. With high availability on, both now run two replicas. A cluster with one master keeps one replica. The rollout keeps one replica running (maxSurge: 0,maxUnavailable: 1). Pod anti-affinity and both PodDisruptionBudgets are unchanged (CSI:maxUnavailable: 1; module controller:minAvailable: 1). The rule lives in a new define,helm_lib_module_controller_on_master_strategy_and_replicas_for_ha, used by both templates. A module controller withonMasterNode: falsealready ran two replicas.attacherEnabled(defaulttrue). Whenfalse, the external-attacher container and its VPA container policy are not rendered. Meant for drivers whose CSIDriver hasattachRequired: false(csi-nfs, csi-s3, cephfs in csi-ceph): Kubernetes creates no VolumeAttachment for them and the attacher runs idle.resizerHandleVolumeInUseError(defaulttrue). Whenfalse, the external-resizer gets--handle-volume-inuse-error=false.Both new options render exactly as before when they are not set. The chart version goes to 1.72.35.
Pods of the CSI and module controllers are recreated once when a module picks up the new lib-helm. In HA mode one replica of each is removed. No other cluster components restart.
Why do we need it, and what problem does it solve?
On master nodes, most of the memory of the storage modules goes to process count rather than to any single process: almost every container sits at 15–25 MiB, the floor of a Go process, and each one gets its own memory request from the VPA. Measured on two test clusters with nine storage modules from
main(three 12 GiB masters, DKP v1.74.18 and v1.76.11), the storage containers on the masters peak at 5.3 GiB of working set in total, while their requests after the VPA are:minAllowed25Mi is the floorOn v1.76.11 the masters were at 99–101% of allocatable memory by requests, and 25 storage containers stayed
PendingwithInsufficient memory.helm_lib_module_controller_manifestsare leader-elected too. Their third replica is a standby controller plus its webhooks and kube-rbac-proxy containers.--handle-volume-inuse-error=true, the external-resizer starts an informer over every Pod in the cluster to wait until in-use volumes are released (source). A driver that expands volumes online never reports a volume in use, but its leader resizer still holds that cache: about 60 MiB against 16 MiB on a standby on a test cluster, growing with the number of pods. The flag exists in every resizer we ship, from v1.8.0 up to v2.1.0.The modules opt in to the last two options in their own MRs after this release.
What is the expected result?
csi-controllerruns two pods; with one master, one pod. Without HA mode, one pod withRecreate, as before.attacherEnabled: falsehas noattachercontainer incsi-controllerand noattacherpolicy in its VerticalPodAutoscaler.resizerHandleVolumeInUseError: falsestarts the resizer with--handle-volume-inuse-error=false, and the resizer log has noRegister Pod informer for resizerline.Checklist