feat(snapshot-controller): move to the home-operations OCI chart

[axeII]

Part 3 of #3828. Draft on purpose. Do not merge until the pre-merge sequence below has been run and verified.

This upgrade would delete the six volume-snapshot CRDs unless it is sequenced correctly. The hazard is real, the fix is small, and nothing in CI can see either — details and evidence below.

What changed

• New kubernetes/apps/volsync-system/snapshot-controller/app/repository.yaml — OCIRepository for oci://ghcr.io/home-operations/charts/snapshot-controller, using the same layerSelector mediaType / operation: copy pattern as tuppr and volsync.
• helmrelease.yaml — chart.spec → chartRef.
• upgrade.crds: CreateReplace → Skip — transitional, for this migration only. This is the safety fix; see "Why CreateReplace does not save you". install.crds stays CreateReplace (correct for a fresh install).
• Values ported:
  • controller.serviceMonitor.create: true → monitoring.serviceMonitor.enabled: true (key moved).
  • controller.volumeGroupSnapshots: true — set explicitly; the chart default is already true, so this is a no-op that documents intent and keeps the feature gate identical to the piraeus release.
  • resources added (chart default is {}): cpu: 10m / memory: 64Mi requests, memory: 128Mi limit. resources is top-level in this chart, not under controller.
• kubernetes/flux/meta/repos/piraeus.yaml deleted and dropped from the repos kustomization. grep -rn piraeus kubernetes/ returns nothing, and this was the only HelmRelease using that source.

Chart version is 0.1.1, not the 0.1.0 the issue names. 0.1.1 was published 2026-09-09; the only change is a vendir lockfile maintenance bump. appVersion is v8.6.0 in both.

The hazard

The two charts ship the same CRDs by different mechanisms:

Chart	CRD mechanism	In the release manifest?
piraeus 5.2.0	templates/crds.yaml, gated on installCRDs (default true)	yes
home-operations 0.1.1	crds/ directory	no — Helm excludes crds/

Confirmed against the live release: the stored manifest for sh.helm.release.v1.snapshot-controller.v9 contains 6 kind: CustomResourceDefinition, all sourced from snapshot-controller/templates/crds.yaml. Flux's .status.inventory lists all six. None of the live CRDs carries helm.sh/resource-policy: keep; all carry meta.helm.sh/release-name: snapshot-controller.

helm upgrade deletes everything present in the old release manifest and absent from the new one. So the delete set for this upgrade is the six CRDs plus the four renamed RBAC objects (harmless).

Deleting those CRDs cascades to every VolumeSnapshot and VolumeSnapshotContent in the cluster. Current inventory: 12 VolumeSnapshots and 12 VolumeSnapshotContents, all deletionPolicy: Delete, ~68 GiB of Ceph RBD snapshots on rook-ceph.rbd.csi.ceph.com. They are the status.latestImage of the 12 ReplicationDestinations. Because the controller is still running during its own upgrade, it would process each deletion and instruct the Ceph CSI driver to delete the underlying RBD snapshots.

It also takes both VolumeSnapshotClasses (csi-ceph-blockpool, csi-ceph-filesystem) with it. Their own helm.sh/resource-policy: keep does not help — deleting a CRD destroys its instances regardless. They belong to the rook-ceph-cluster release under a post-install,pre-upgrade hook, so they would only return on the next rook-ceph HelmRelease upgrade, not on a plain reconcile.

Scope of the damage, stated honestly: the kopia repositories are untouched, so no backup history is lost. What is lost is the 12 pre-staged restore images, rebuildable only by re-running all 12 ReplicationDestinations. Nightly backups for the 12 volsync apps would also stop until the CRDs are restored — VolSyncVolumeOutOfSync (critical, 5m) would fire.

Why CreateReplace does not save you

This is the part that is easy to get wrong, and it is why this PR sets Skip.

Helm skips deleting a resource whose live object carries helm.sh/resource-policy: keep, so annotating the CRDs beforehand looks sufficient. On this cluster it is not, because Flux's own CRD step runs first and removes the annotation:

1. helm-controller runs applyCRDs() before the Helm upgrade — internal/action/upgrade.go:85, then :91.
2. serverSideApply resolves from the last release: serverSideApply = lastReleaseTyped.ApplyMethod == "ssa" (upgrade.go:74). Release v9 records apply_method: csa, so this is false.
3. With SSA off, CreateReplace adds ClientUpdateOptionForceReplace(true) (internal/action/crds.go:225-227).
4. Force-replace is a full PUT of the chart's CRD file (helm pkg/kube/client.go:839 → :1143). The chart's crds/*.yaml carry no resource-policy annotation, so the PUT strips the keep annotation.
5. Helm's upgrade then reaches its delete loop (client.go:654), reads the live object (:678), finds no keep, and deletes.

Annotation applied → stripped by Flux → checked by Helm. The CRDs go anyway.

Versions this was verified against: helm-controller v1.6.4, Helm SDK v4.2.4, Flux v2.9.5. Under server-side apply the annotation would survive, which is why apply_method: csa is the decisive fact — do not assume this generalises to a release that was applied server-side.

upgrade.crds: Skip closes it: applyCRDs returns before touching anything (crds.go:56), so the annotation is still there when Helm checks. This costs nothing — see the CRD equivalence check below.

The Service is an accidental safety net — do not remove it early

The old chart made the metrics Service headless; the new one renders a normal ClusterIP Service with no clusterIP field. Live object and the stored v9 manifest both pin clusterIP: None. spec.clusterIP is immutable, so Helm's patch is rejected with a "field is immutable" error.

Helm aborts Update() on the first patch error (client.go:651) before reaching the delete loop (:654). So on an unprepared cluster this failure would accidentally spare the CRDs.

That is ordering luck, not a safeguard, and it matters for sequencing: deleting the Service is what arms the CRD deletion. Doing the Service fix without the CRD protection actually in place is worse than doing neither. Hence the order below.

Only the ServiceMonitor consumes this Service and it selects by label, so deleting it is harmless; Helm recreates it non-headless.

Pre-merge sequence — run in this order

1. Protect the CRDs.

for crd in \
  volumesnapshots.snapshot.storage.k8s.io \
  volumesnapshotcontents.snapshot.storage.k8s.io \
  volumesnapshotclasses.snapshot.storage.k8s.io \
  volumegroupsnapshots.groupsnapshot.storage.k8s.io \
  volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io \
  volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io; do
  kubectl annotate crd "$crd" helm.sh/resource-policy=keep --overwrite
done

2. Verify all six — do not skip this.

kubectl get crd -o custom-columns=\
'NAME:.metadata.name,KEEP:.metadata.annotations.helm\.sh/resource-policy' \
  | grep -E 'volumesnapshot|volumegroupsnapshot'

Every row must read keep. If any says <none>, stop.

3. Back up the CRD manifests. Cheap insurance, and with the specs identical this makes even the worst case a one-command recovery.

kubectl get crd -o yaml \
  volumesnapshots.snapshot.storage.k8s.io \
  volumesnapshotcontents.snapshot.storage.k8s.io \
  volumesnapshotclasses.snapshot.storage.k8s.io \
  volumegroupsnapshots.groupsnapshot.storage.k8s.io \
  volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io \
  volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io \
  > snapshot-crds-backup.yaml

4. Delete the headless Service — only now that step 2 has passed.

kubectl delete svc snapshot-controller -n volsync-system

5. Merge, then watch the release:

flux -n volsync-system get helmrelease snapshot-controller --watch

Optional, if you want proof before committing to it: merge with the Service still in place. The upgrade fails on the immutable clusterIP and the CRDs stay untouched, which demonstrates the protection is holding. It will churn through remediation.retries: 3 with rollbacks, so it is noisy — then do step 4 and flux reconcile hr snapshot-controller -n volsync-system.

Backups last ran cleanly at 02:00 today, so the kopia repositories are current; a manual ReplicationSource trigger first is optional rather than necessary.

Post-merge checks

kubectl get crd | grep -E 'volumesnapshot|volumegroupsnapshot'   # all six present
kubectl get volumesnapshot -A                                     # still 12, all READYTOUSE=true
kubectl get volumesnapshotcontent | wc -l                         # still 12
kubectl get volumesnapshotclass                                   # csi-ceph-blockpool, csi-ceph-filesystem
kubectl -n volsync-system rollout status deploy/snapshot-controller

Then trigger one ReplicationSource manually to confirm end to end, and check the next nightly run at 02:00.

Follow-up

Restore upgrade.crds: CreateReplace in a separate PR once this has landed. At that point the CRDs are in neither the old nor the new release manifest, so Helm has nothing to delete and the force-replace is harmless. Leaving Skip in place permanently would mean future CRD updates from the chart never apply.

Why CI cannot catch any of this

flate renders zero CRDs on both sides — verified by rendering origin/main and this branch. The piraeus CRDs come from templates/, yet they do not appear in flate's output either.

Konflate's summary (+5 added · 5 changed · −5 removed) accounts exactly for the four RBAC renames plus the repo swap, the five added counterparts, and the five changed objects. No CRD appears anywhere in the diff.

A renderer compares rendered output. The deletion here comes from Helm's manifest diff against the stored release, which no offline renderer models. The whole validation chain is structurally blind to this class of change — worth remembering for future templates/ → crds/ migrations.

What was verified

• CRD content is equivalent. Rendered the piraeus templates/crds.yaml with this repo's values and diffed all six .spec blocks against the new chart's crds/*.yaml: byte-identical. Same controller-gen v0.15.0. Both leave conversion unset (webhook.enabled: false), and the live CRDs report conversion.strategy: None. Storage versions match: v1 for the three snapshot CRDs, v1beta2 for the three group CRDs. Nothing is lost by not applying them, which is what makes Skip free.
• No group-snapshot objects exist — 0 VolumeGroupSnapshots, 0 contents, 0 classes. The feature gate is preserved but unused.
• Values validate against the chart's values.schema.json; rendered args are unchanged in substance: --v=2 --leader-election=true --leader-election-namespace=$(NAMESPACE) --http-endpoint=:8080 --feature-gates=CSIVolumeGroupSnapshot=true.
• volsync is healthy right now — all 11 ReplicationSources synced 02:00–02:01 today, next run 2026-09-13T02:00. (media/overseerr has a destination but no source; pre-existing, unrelated.)
• Nothing but volsync uses VolumeSnapshots — all 12 are labelled app.kubernetes.io/created-by: volsync with a single ReplicationDestination owner; no PVC anywhere has a dataSource of kind VolumeSnapshot.

Rendered diff

Names stay put — Deployment, Service, ServiceAccount and ServiceMonitor remain snapshot-controller, and the Deployment's selector.matchLabels is unchanged, so this is an in-place upgrade with nothing orphaned.

RBAC is renamed (Helm creates the new and removes the old within the release; rules are byte-identical):

Kind	Before	After
ClusterRole / ClusterRoleBinding	snapshot-controller	snapshot-controller-runner
Role / RoleBinding	snapshot-controller	snapshot-controller-leaderelection

The RoleBinding subject also gains an explicit namespace: volsync-system instead of relying on defaulting.

Deployment changes:

• Container port and probe target renamed http → metrics (same port 8080).
• Probes gain initialDelaySeconds / periodSeconds; args gain --v=2 and are reordered.
• resources now set (was {}). Peak working set over the last 7d is 28 MiB.
• runAsUser 1000 → 65532, plus a pod-level securityContext (fsGroup/runAsGroup 65532, seccompProfile: RuntimeDefault), allowPrivilegeEscalation: false and readOnlyRootFilesystem: true. The controller writes nothing to disk.
• Adds priorityClassName: system-cluster-critical, enableServiceLinks: false, minReadySeconds: 35, and a maxSurge: 0 / maxUnavailable: 1 strategy — so the single replica is down for roughly 40–60s during rollout. Harmless outside the 02:00 window.
• ServiceMonitor gains explicit interval: 30s / scrapeTimeout: 10s.

Validation

• just configure — clean
• just validate (yayamlls) — clean
• just flate-test — 169 passed, including OCIRepository volsync-system/snapshot-controller; the one warning (external-dns-cloudflare: values not used by the chart) is pre-existing and unrelated
• python3 scripts/find_mistakes.py — exits 0, only the two known clean-tree warnings
• pre-commit run --all-files — all hooks passed

[bot-akira]

Warning

• ClusterRoleBinding snapshot-controller-runner: new ClusterRoleBinding; grants cluster-wide permissions
• CustomResourceDefinition volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• CustomResourceDefinition volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• CustomResourceDefinition volumegroupsnapshots.groupsnapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• CustomResourceDefinition volumesnapshotclasses.snapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• CustomResourceDefinition volumesnapshotcontents.snapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• CustomResourceDefinition volumesnapshots.snapshot.storage.k8s.io: removed CustomResourceDefinition; deletes all of its custom resources
• snapshot-controller: major chart version bump 5.2.0 → 0.1.1; check the chart's upgrade notes for breaking changes

Note

+5 added · 5 changed · −11 removed: 21 resources across 3 apps · 6 CRDs

Blast radius

• Kustomization flux-system/cluster-meta: 1 dependent (flux-system/cluster-apps)

_{konflate 0.6.4 · rendered dafd769 · full diff →}

[axeII]

session.md

Since I am not sure when I'll get chance to take a look at this and this is a low priority here is handoff document