Incremental nas backup fixes#13571
Conversation
…- Backup deletes are serialized by using a per-VM GlobalLock Backup delete checks if it has a live dependent backup not just a live immediate child.
|
HI @jmsperu |
There was a problem hiding this comment.
Pull request overview
This PR builds on the incremental NAS backup work by hardening incremental chain deletion: it serializes chain-mutating deletes with a per-VM lock, fixes the “can I delete this backup?” decision to consider all downstream dependents (not just the immediate child), removes dead chain-navigation helpers, and adds a null-check around answer.getNeedsCleanup().
Changes:
- Serialize backup-chain deletions per VM using a
GlobalLockand re-read the target backup row under the lock to avoid race-induced “stuck Hidden” backups. - Update chain delete semantics to detect any live descendants (via
CHAIN_POSITION) before physically deleting a backup; enhance sweep behavior and add multiple new unit tests for edge cases. - Remove unused chain helper methods and add a null-check before calling
answer.getNeedsCleanup().
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java |
Adds per-VM locking for chain deletes and updates descendant detection + sweeping logic for incremental backup chains. |
plugins/backup/nas/src/test/java/org/apache/cloudstack/backup/NASBackupProviderTest.java |
Updates and extends unit tests to cover new lock behavior and additional chain-delete edge cases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@blueorangutan package |
|
@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java:1136
chainPositionreturnsInteger.MAX_VALUEfor missing/invalidCHAIN_POSITION, butgetChainOrderedLeafToRootsorts in descending order. That means missing metadata sorts to the front (treated as the newest/leaf), contradicting the in-code comment and potentially causing incorrect sweep ordering and dependent detection when metadata is missing or malformed.
private int chainPosition(Backup b) {
String s = readDetail(b, NASBackupChainKeys.CHAIN_POSITION);
if (s == null) {
return Integer.MAX_VALUE; // no metadata => sort to end
}
try {
return Integer.parseInt(s);
} catch (NumberFormatException e) {
return Integer.MAX_VALUE;
}
| private boolean hasLiveChildren(Backup backup) { | ||
| String chainId = readDetail(backup, NASBackupChainKeys.CHAIN_ID); | ||
| if (chainId == null) { | ||
| return false; | ||
| } | ||
| int position = chainPosition(backup); | ||
| for (Backup b : backupDao.listByVmId(null, backup.getVmId())) { | ||
| if (parentUuid.equals(b.getUuid())) { | ||
| return b; | ||
| if (b.getId() == backup.getId() || !chainId.equals(readDetail(b, NASBackupChainKeys.CHAIN_ID))) { | ||
| continue; | ||
| } | ||
| if (!isDeletePending(b) && chainPosition(b) > position) { | ||
| return true; |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #13571 +/- ##
======================================
Coverage 3.46% 3.46%
======================================
Files 479 479
Lines 41162 41162
Branches 7793 7793
======================================
Hits 1426 1426
Misses 39543 39543
Partials 193 193
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18501 |
|
@blueorangutan test |
|
@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
Description
This PR contains some fixes that are required on top of #13074
Concurrent deletes can modify the same backup chain in parallel. There is no synchronisation mechanism, so in a race condition a backup can be left in the Hidden state without ever being deleted. Using a per-VM global lock to synchronise deletes.
If a backup can be deleted only checks if its immediate child is present or not (not hidden and not removed). But it should check all its dependents. Otherwise it will be wrongly deleted.
Removed dead code: findChainTail and findChainParent
Added null check before accessing answer.getNeedsCleanup()
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Reproduced the failing test cases and verified with the fixes.
How did you try to break this feature and the system with this change?