Skip to content

feat: trim the CSI and module controller memory footprint - #234

Merged
duckhawk merged 2 commits into
mainfrom
feat/csi-controller-memory-footprint
Nov 15, 2026
Merged

duckhawk merged 2 commits into
mainfrom
feat/csi-controller-memory-footprint

Conversation

@duckhawk

@duckhawk duckhawk commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Description

helm_lib_csi_controller_manifests and helm_lib_module_controller_manifests get changes that cut the number of processes, and with them the memory requests, of the CSI and module controllers on master nodes:

  • Two replicas in HA mode on masters. The CSI controller with csiControllerHaMode: true and the module controller with onMasterNode: true (the default) ran one replica per master. With high availability on, both now run two replicas. A cluster with one master keeps one replica. The rollout keeps one replica running (maxSurge: 0, maxUnavailable: 1). Pod anti-affinity and both PodDisruptionBudgets are unchanged (CSI: maxUnavailable: 1; module controller: minAvailable: 1). The rule lives in a new define, helm_lib_module_controller_on_master_strategy_and_replicas_for_ha, used by both templates. A module controller with onMasterNode: false already ran two replicas.
  • attacherEnabled (default true). When false, the external-attacher container and its VPA container policy are not rendered. Meant for drivers whose CSIDriver has attachRequired: false (csi-nfs, csi-s3, cephfs in csi-ceph): Kubernetes creates no VolumeAttachment for them and the attacher runs idle.
  • resizerHandleVolumeInUseError (default true). When false, the external-resizer gets --handle-volume-inuse-error=false.

Both new options render exactly as before when they are not set. The chart version goes to 1.72.35.

Pods of the CSI and module controllers are recreated once when a module picks up the new lib-helm. In HA mode one replica of each is removed. No other cluster components restart.

Why do we need it, and what problem does it solve?

On master nodes, most of the memory of the storage modules goes to process count rather than to any single process: almost every container sits at 15–25 MiB, the floor of a Go process, and each one gets its own memory request from the VPA. Measured on two test clusters with nine storage modules from main (three 12 GiB masters, DKP v1.74.18 and v1.76.11), the storage containers on the masters peak at 5.3 GiB of working set in total, while their requests after the VPA are:

DKP VPA rounding Requests on 3 masters
1.72–1.75 none, minAllowed 25Mi is the floor 9.5 GiB
1.76.0–1.76.11 up to 64Mi 18.4 GiB
1.76.12+ up to 16Mi 11.0 GiB

On v1.76.11 the masters were at 99–101% of allocatable memory by requests, and 25 storage containers stayed Pending with Insufficient memory.

  • Every CSI sidecar and the driver are leader-elected, so the third replica of an HA controller is one more standby with six to eight containers. Ten drivers run in HA mode.
  • The module controllers of all storage modules that use helm_lib_module_controller_manifests are leader-elected too. Their third replica is a standby controller plus its webhooks and kube-rbac-proxy containers.
  • With the upstream default --handle-volume-inuse-error=true, the external-resizer starts an informer over every Pod in the cluster to wait until in-use volumes are released (source). A driver that expands volumes online never reports a volume in use, but its leader resizer still holds that cache: about 60 MiB against 16 MiB on a standby on a test cluster, growing with the number of pods. The flag exists in every resizer we ship, from v1.8.0 up to v2.1.0.

The modules opt in to the last two options in their own MRs after this release.

What is the expected result?

  • In HA mode on a cluster with two or more masters, csi-controller runs two pods; with one master, one pod. Without HA mode, one pod with Recreate, as before.
  • The module controller on masters runs two pods in HA mode on a cluster with two or more masters, one pod with one master or without HA, as before.
  • A module with attacherEnabled: false has no attacher container in csi-controller and no attacher policy in its VerticalPodAutoscaler.
  • A module with resizerHandleVolumeInUseError: false starts the resizer with --handle-volume-inuse-error=false, and the resizer log has no Register Pod informer for resizer line.
  • Modules that set neither option render the same containers and arguments as before.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

The CSI controller is one of the largest consumers of memory requests on
master nodes: every replica runs the driver and five sidecars, and each of
those processes gets its own memory request from the VPA. Measured on
DKP 1.74.18 and 1.76.11 with nine storage modules, the sidecars and drivers
on the three masters peak at 5.3 GiB of working set while their requests
reach 9.5 GiB (1.72-1.75), 18.4 GiB (1.76.0-1.76.11, 64Mi rounding) and
11.0 GiB (1.76.12 and later).

- In HA mode the controller now runs two replicas instead of one per
  master. Every sidecar is leader-elected, so the third replica only adds
  another standby. A cluster with one master keeps one replica, and the
  rollout keeps one replica running (maxSurge 0, maxUnavailable 1).
- attacherEnabled (default true) lets a driver whose CSIDriver has
  attachRequired: false drop the external-attacher container and its VPA
  policy. Kubernetes creates no VolumeAttachment for such a driver, so the
  attacher has nothing to do.
- resizerHandleVolumeInUseError (default true) passes
  --handle-volume-inuse-error=false to the external-resizer. With the
  upstream default the resizer starts an informer over every Pod in the
  cluster to wait for in-use volumes to be released; a driver that expands
  volumes online never reports them, and the leader resizer then holds a
  cache that grows with the number of pods (about 60 MiB against 16 MiB on
  a standby). The flag exists in every resizer version we ship (v1.8.0
  and later).

Both new options keep the current rendering when unset.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
helm_lib_module_controller_manifests with onMasterNode ran one replica per
master in HA mode, like the CSI controller did. The controller is
leader-elected, so with three masters the third replica only adds a
standby, along with its webhooks and kube-rbac-proxy containers and their
memory requests.

Move the two-replica rule from the CSI controller into
helm_lib_module_controller_on_master_strategy_and_replicas_for_ha and use
it for both: two replicas in HA mode, one on a cluster with a single
master, maxSurge 0 and maxUnavailable 1. Without HA the rendering is
unchanged. A module controller off masters already ran two replicas.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@duckhawk duckhawk changed the title feat(csi): trim the CSI controller memory footprint feat: trim the CSI and module controller memory footprint Nov 15, 2026
@kkozoriz
kkozoriz self-requested a review November 15, 2026 10:49
@duckhawk
duckhawk merged commit ee2919f into main Nov 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants