Bugfix/cstackex 327-Concurrent iSCSI VM deployment fails intermittently: post-clone LUN resize returns 404 by sp16743 · Pull Request #118 · NetApp/cloudstack

sp16743 · 2026-10-09T16:49:51Z

Description

After an iSCSI template clone, CloudStack grows the new LUN when the service offering disk is larger than the template. ONTAP returns the LUN UUID from the clone, but the immediate resize PATCH /api/storage/luns/{uuid} can return HTTP 404 (entry doesn't exist, target uuid) before that LUN is visible.

resizeCloudStackVolume now retries only that PATCH, using the same UUID. A 404 is retried for up to 30 attempts, one second apart. Any other error fails on the first attempt.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

Before the fix:

How Has This Been Tested?

Live concurrent deployment:
-Created an ONTAP iSCSI primary storage pool.
-Deployed ten CentOS VMs concurrently using the ISCSI compute offering.
-Nine deployments reached LUN cloning; one failed earlier due to insufficient host capacity.
-Nine clone POST requests returned LUN UUIDs.
-Five cloned LUNs were visible on the first GET.
-Four cloned LUNs returned 404 on the first GET.
-For all four, CloudStack waited and retried; the LUN became visible on attempt 2/5.
-CloudStack then resized each LUN to 10 GiB using PATCH and continued VM deployment successfully.
-No clone failed with Cloned Lun uuid was not visible.

How did you try to break this feature and the system with this change?

github-actions · 2026-10-09T17:54:16Z

🔴 Test Coverage Grade: `D` — Marginal

Metric	Value
Line coverage	24.82%
Branch coverage	18.97%

Grade Scale

Grade	Line Coverage	Meaning
🟢 A	≥ 80%	Excellent - this code sleeps well at night 😴
🟡 B	60-79%	Good - almost there, don't stop now 😉
🟠 C	40-59%	Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D	20-39%	Marginal - boldly shipping where no test has gone before 🖖
⛔ F	< 20%	Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

github-actions · 2026-10-12T05:29:50Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

suryag1201 · 2026-10-12T19:07:43Z

        if (sizeInBytes <= 0) {
            throw new CloudRuntimeException("Failed to resize Lun, invalid size " + sizeInBytes);
        }
+        String authHeader = OntapStorageUtils.generateAuthHeader(storage.getUsername(), storage.getPassword());


instead of adding the retry logic here, during lun clone return the job response and poll for that job to get complete and perform the resize later

github-actions · 2026-10-26T15:05:58Z

🔴 Test Coverage Grade: `D` — Marginal

Metric	Value
Line coverage	25.83%
Branch coverage	20.12%

Grade Scale

Grade	Line Coverage	Meaning
🟢 A	≥ 80%	Excellent - this code sleeps well at night 😴
🟡 B	60-79%	Good - almost there, don't stop now 😉
🟠 C	40-59%	Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D	20-39%	Marginal - boldly shipping where no test has gone before 🖖
⛔ F	< 20%	Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

sandeeplocharla · 2026-11-04T09:50:45Z

+                }
+            }
+        }
+        throw new CloudRuntimeException("Cloned Lun uuid was not visible for: " + lunUuid);


If lun uuid itself is not visible, what's the point of logging lunUuid? 😆

Yes, there is no point. I have changed it.

sandeeplocharla

LGTM 🚀

suryag1201 · 2026-10-29T06:14:21Z

            String authHeader = OntapStorageUtils.generateAuthHeader(storage.getUsername(), storage.getPassword());
-            OntapResponse<Lun> clonedLun = sanFeignClient.createLun(authHeader, true, lunRequest);
-            if (clonedLun == null || CollectionUtils.isEmpty(clonedLun.getRecords())) {
+            OntapResponse<Lun> created = sanFeignClient.createLun(authHeader, true, lunRequest);


change the variable name to clonedLun

suryag1201 · 2026-10-29T06:15:08Z

-            OntapResponse<Lun> clonedLun = sanFeignClient.createLun(authHeader, true, lunRequest);
-            if (clonedLun == null || CollectionUtils.isEmpty(clonedLun.getRecords())) {
+            OntapResponse<Lun> created = sanFeignClient.createLun(authHeader, true, lunRequest);
+            if (created == null || created.getRecords() == null || created.getRecords().isEmpty()


use collectionUtils/Stringutils for null or empty check

Changed it now

suryag1201 · 2026-11-12T05:22:27Z

            }
-            Lun lun = clonedLun.getRecords().get(0);
+            String lunUuid = created.getRecords().get(0).getUuid();
+            Lun lun = getLunByUuidUntilVisible(authHeader, lunUuid, lunRequest.getName());


what if the get lun call failed, cloned lun will. still exists and retry will see the duplicate lun

Added it.The cloned LUN is deleted ,so a subsequent CloudStack retry does not encounter a duplicate LUN name.

suryag1201 · 2026-11-12T05:28:28Z

+        for (int attempt = 1; attempt <= OntapStorageConstants.ONTAP_LUN_CLONE_GET_MAX_RETRIES; attempt++) {
+            try {
+                Lun lun = sanFeignClient.getLunByUUID(authHeader, lunUuid);
+                if (lun != null) {


we can not decode based on lun != null condition that lun is visible.
return the get response record instead

Changed it.The code now uses the collection GET with the LUN UUID, verifies that the response contains a record, and returns that record instead of relying on lun != null.

suryag1201 · 2026-11-12T05:29:48Z

    public static final int ONTAP_FLEXVOL_JOB_POLL_INTERVAL_MS = 2000;
    public static final int ONTAP_FLEXVOL_RESOLVE_MAX_RETRIES = 30;
+    /** GET-by-UUID retries after LUN clone response. */
+    public static final int ONTAP_LUN_CLONE_GET_MAX_RETRIES = 5;


this interval and no of retries are less for clone

Changed the retries to 10

sp16743 requested review from piyush5netapp, rajiv-jain-netapp, sandeeplocharla and suryag1201 as code owners October 9, 2026 16:49

github-actions Bot added the status:has-conflicts label Oct 12, 2026

suryag1201 reviewed Oct 12, 2026

View reviewed changes

suryag1201 requested changes Oct 12, 2026

View reviewed changes

Fix ONTAP LUN clone visibility using GET retry

182b07b

sp16743 force-pushed the bugfix/CSTACKEX-327 branch from 62c8ec7 to 182b07b Compare October 26, 2026 14:00

github-actions Bot removed the status:has-conflicts label Oct 26, 2026

sandeeplocharla reviewed Nov 4, 2026

View reviewed changes

Change the cloned LUN read failure message

a0586de

sandeeplocharla approved these changes Nov 7, 2026

View reviewed changes

suryag1201 reviewed Nov 12, 2026

View reviewed changes

sp16743 added 2 commits November 12, 2026 18:45

Addressing Comments

99194b7

Restore unused LUN endpoint

e68f745

Conversation

sp16743 commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

github-actions Bot commented Oct 9, 2026

🔴 Test Coverage Grade: D — Marginal

Grade Scale

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test Coverage Grade: D — Marginal

Grade Scale

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeeplocharla left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sp16743 commented Oct 9, 2026 •

edited

Loading

🔴 Test Coverage Grade: `D` — Marginal

github-actions Bot commented Oct 26, 2026 •

edited

Loading

🔴 Test Coverage Grade: `D` — Marginal