From e9f8cc8bad9f5c162f4331431ccfedfd9e3d0883 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 13 Nov 2026 17:02:19 +0300 Subject: [PATCH] docs(clickhouse): describe deleting a ClickHouse archive with its Backup cozystack/cozystack#5224 makes deleting a Backup taken on the system bucket delete its clickhouse-backup archive, and the trunk docs still said the archive stays and is governed by the sidecar's retention, which the chart does not configure. The backup classes page described the cleanup of the other drivers but not of ClickHouse, which now has its own Events, retry and release cases. The v1.4 page sent administrators to DELETE /backup//remote, which clickhouse-backup 2.7 does not serve; the route is POST /backup/delete/remote/. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../en/docs/next/applications/backup-and-recovery.md | 4 ++-- .../en/docs/next/operations/services/backup-classes.md | 10 ++++++++++ .../services/managed-app-backup-configuration.md | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/content/en/docs/next/applications/backup-and-recovery.md b/content/en/docs/next/applications/backup-and-recovery.md index 7395adf4a..05c7cfec7 100644 --- a/content/en/docs/next/applications/backup-and-recovery.md +++ b/content/en/docs/next/applications/backup-and-recovery.md @@ -183,10 +183,10 @@ The source application stays untouched. Cross-namespace restores are **not** sup ## Limitations and lifecycle - **Data-only scope.** Application CRs, HelmReleases, chart values, and operator-managed Secrets (e.g. `cnpg.io` superuser secret, `clickhouse-installation` users) are not captured. Pre-provision the target application before a to-copy restore. -- **Archive retention is driver-owned.** Deleting a Cozystack `Backup` CR removes the artefact reference but leaves the actual S3 object intact. Each driver enforces its own retention: +- **Archive retention is driver-owned.** Deleting a Cozystack `Backup` CR removes the artefact reference; whether the S3 object goes with it depends on the driver: - CNPG: `retentionPolicy` on the strategy (admin-owned; default `30d` in the admin example). - MariaDB: `cleanupStrategy` on the operator-side `Backup` CR or rotation at the bucket level (admin-owned). - - ClickHouse: governed by the in-pod sidecar's retention configuration. To purge an archive ahead of schedule, ask your administrator — the call goes against the `clickhouse-backup` HTTP API on the sidecar. + - ClickHouse: on `backup.useSystemBucket: true`, deleting the `Backup` deletes its archive through the in-pod `clickhouse-backup` sidecar, and the `Backup` stays until that delete finished; an archive it cannot or must not delete is named in an `ArtifactNotDeleted` Warning Event on the `Backup` (see [ClickHouse archive cleanup]({{% ref "/docs/next/operations/services/backup-classes#clickhouse-archive-cleanup" %}})). On the application's own bucket the archive stays until removed through the sidecar's `POST /backup/delete/remote/` or the bucket's lifecycle rules. - **ClickHouse depends on the in-chart sidecar.** The Altinity strategy is a thin HTTP client; the backup itself runs inside each `chi-*` Pod via `clickhouse-backup`. Disabling `backup.enabled` on the application also disables the BackupClass flow. ## Troubleshooting diff --git a/content/en/docs/next/operations/services/backup-classes.md b/content/en/docs/next/operations/services/backup-classes.md index 8c9e1f8dd..dd91df5c2 100644 --- a/content/en/docs/next/operations/services/backup-classes.md +++ b/content/en/docs/next/operations/services/backup-classes.md @@ -111,6 +111,16 @@ When `useSystemBucket: true`: `s3Region`, `s3Bucket`, `endpoint`, `s3AccessKey`, `s3SecretKey`, and `s3CredentialsSecret` are ignored in this mode. +### ClickHouse archive cleanup + +On `backup.useSystemBucket: true` the driver owns the archive, which nothing else prunes in the shared bucket. Each backup names its archive after the run (`--`) and records it on the `Backup`, together with the `useSystemBucket` setting and the system bucket the projected `cozy-backups-creds` named when the backup Job was created. Deleting such a `Backup` runs a one-shot cleanup Job that calls `POST /backup/delete/remote/` on the application's sidecar, and the `Backup` stays until that delete finished. An archive already gone counts as deleted. + +The cleanup Job waits out a sidecar that is busy with another operation or restarting, and starts only once changed credentials have reached the sidecar. A delete that still fails is reported as an `ArtifactCleanupFailed` Warning Event on the `Backup` and retried after a wait that starts at 30 seconds and doubles with each failure, up to 30 minutes. The cleanup Pod runs the strategy container's image with its pull settings and is scheduled like the strategy's own Pods (node selector, tolerations, affinity, priority class). + +The `Backup` is released without deleting the archive, with an `ArtifactNotDeleted` Warning Event naming the archive left in the bucket, when the namespace is terminating; the strategy, the application or its `clickhouse--backup-api-auth` Secret is gone; the application no longer uses the system bucket, or the system bucket is no longer the one the `Backup` records or cannot be read (also when either changes while the delete runs, in which case the Event says the archive may remain); another Job holds the cleanup Job name; or the archive is named in the older `clickhouse--` form, which two runs started in the same second could share. The `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation releases a `Backup` without starting a delete; a delete the sidecar already accepted still runs to the end. + +A `Backup` that records no archive name, as one taken by a strategy that does not use `.ArchiveName` does, or no bucket setting, as one taken before this behaviour does, is released without an Event, leaving its archive. On the application's own bucket the archive's lifecycle stays the tenant's: delete it through the sidecar's `POST /backup/delete/remote/` or with the bucket's lifecycle rules. + ## MongoDB: backup storage MongoDB supports two storage flows. By default the S3 target lives on the app CR and the tenant supplies its own credentials; alternatively the tenant opts into the platform system bucket and the driver injects the storage onto the live cluster at backup time. diff --git a/content/en/docs/v1.4/operations/services/managed-app-backup-configuration.md b/content/en/docs/v1.4/operations/services/managed-app-backup-configuration.md index 45cca86dc..66cb050c5 100644 --- a/content/en/docs/v1.4/operations/services/managed-app-backup-configuration.md +++ b/content/en/docs/v1.4/operations/services/managed-app-backup-configuration.md @@ -334,4 +334,4 @@ kubectl -n tenant-user get backups.k8s.mariadb.com,restores.k8s.mariadb.com kubectl -n tenant-user logs -l backups.cozystack.io/owned-by.BackupJobName=my-clickhouse-adhoc ``` -For ClickHouse archive purges, the tenant cannot reach the in-pod `clickhouse-backup` sidecar HTTP API directly; on their request, exec into the ClickHouse pod and call `DELETE /backup//remote` against the local sidecar (the chart-emitted `clickhouse--backup-api-auth` Secret carries the credentials). +For ClickHouse archive purges, the tenant cannot reach the in-pod `clickhouse-backup` sidecar HTTP API directly; on their request, exec into the ClickHouse pod and call `POST /backup/delete/remote/` against the local sidecar (the chart-emitted `clickhouse--backup-api-auth` Secret carries the credentials).