Environment
- CloudNativePG operator: 1.26.0
- plugin-barman-cloud: v0.6.0
- PostgreSQL: 17.5
- Restore path:
bootstrap.recovery (externalClusters + barman-cloud plugin), object storage on Azure Blob, restore PVC ~50Gi for a ~8GB source database
What happened
A bootstrap recovery Job failed repeatedly against a backup with a WAL-archiving gap (separate report filed against cloudnative-pg/cloudnative-pg about the Job always retrying the same pinned backup). Kubernetes' default Job.spec.backoffLimit: 6 gave it 7 total pod attempts. The first 6 attempts each:
- Ran
barman-cloud-restore successfully, extracting the full base backup (~8GB) into PGDATA on the restore PVC.
- Started PostgreSQL in recovery, failed WAL replay, and exited.
Nothing cleaned PGDATA between these attempts. Each retry re-extracted the same ~8GB base backup on top of / alongside whatever the previous attempt left behind, so usage accumulated across attempts. By the 7th attempt, the ~50Gi restore PVC was full, and that attempt failed differently and much earlier:
ERROR: Barman cloud restore exception: [Errno 28] No space left on device
Why this is a problem
This turns a clear, correctly-diagnosable failure (a WAL-archiving gap, "WAL ends before end of online backup") into a confusing, unrelated-looking failure (disk exhaustion) purely as a side effect of retrying without cleanup. Whoever investigates the final Job state sees the disk-space error, not the real cause, unless they dig through every earlier retry pod's logs individually. It also means a PVC sized correctly for the database itself is not necessarily sized correctly for backoffLimit + 1 retries of it.
Suggestion
Clean/wipe the target PGDATA directory (and pgdata-adjacent restore artifacts) at the start of each restore attempt, before barman-cloud-restore re-extracts the base backup — or at minimum, detect and fail fast if PGDATA is non-empty going into a retry, rather than silently extracting on top of leftover data from a previous failed attempt.
Happy to provide more logs/detail if useful. This was observed in a production DR setup, not a lab reproduction, so some specifics have been generalized above.
Environment
bootstrap.recovery(externalClusters + barman-cloud plugin), object storage on Azure Blob, restore PVC ~50Gi for a ~8GB source databaseWhat happened
A bootstrap recovery Job failed repeatedly against a backup with a WAL-archiving gap (separate report filed against cloudnative-pg/cloudnative-pg about the Job always retrying the same pinned backup). Kubernetes' default
Job.spec.backoffLimit: 6gave it 7 total pod attempts. The first 6 attempts each:barman-cloud-restoresuccessfully, extracting the full base backup (~8GB) intoPGDATAon the restore PVC.Nothing cleaned
PGDATAbetween these attempts. Each retry re-extracted the same ~8GB base backup on top of / alongside whatever the previous attempt left behind, so usage accumulated across attempts. By the 7th attempt, the ~50Gi restore PVC was full, and that attempt failed differently and much earlier:Why this is a problem
This turns a clear, correctly-diagnosable failure (a WAL-archiving gap, "WAL ends before end of online backup") into a confusing, unrelated-looking failure (disk exhaustion) purely as a side effect of retrying without cleanup. Whoever investigates the final Job state sees the disk-space error, not the real cause, unless they dig through every earlier retry pod's logs individually. It also means a PVC sized correctly for the database itself is not necessarily sized correctly for
backoffLimit + 1retries of it.Suggestion
Clean/wipe the target
PGDATAdirectory (and pgdata-adjacent restore artifacts) at the start of each restore attempt, beforebarman-cloud-restorere-extracts the base backup — or at minimum, detect and fail fast ifPGDATAis non-empty going into a retry, rather than silently extracting on top of leftover data from a previous failed attempt.Happy to provide more logs/detail if useful. This was observed in a production DR setup, not a lab reproduction, so some specifics have been generalized above.