[bernardodemarco]

Description

PR #10140 introduced the allocated_size column to the cloud.object_store table, which is expected to hold the sum of all bucket quota, in bytes. To initialize the new column, the following query was introduced:

cloudstack/engine/schema/src/main/resources/META-INF/db/schema-42040to42100.sql

Lines 742 to 749 in f8f3848

	UPDATE `cloud`.`object_store`
	JOIN (
	SELECT object_store_id, SUM(quota) AS total_quota
	FROM `cloud`.`bucket`
	WHERE removed IS NULL
	GROUP BY object_store_id
	) buckets_quota_sum_view ON `object_store`.id = buckets_quota_sum_view.object_store_id
	SET `object_store`.allocated_size = buckets_quota_sum_view.total_quota;

However, bucket quotas are stored in GiB, not bytes, and the query does not convert them. As a result, Object Storage providers in environments upgraded from versions prior to 4.21 have an allocated_size that is smaller than expected. When a bucket is deleted, its quota (converted to bytes) is subtracted from this value, which becomes negative. Since the column is unsigned, the update fails and the bucket cannot be deleted:

Impossibility of deleting buckets

2026-11-10 16:10:07,939 ERROR [o.a.c.s.d.d.ObjectStoreDaoImpl] (qtp1404565079-3736:[ctx-3a285296, ctx-ce4f3b14]) (logid:78d41388) DB Exception on: HikariProxyPreparedStatement@1008598898 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[21]: UPDATE object_store SET object_store.allocated_size=-536870912000 WHERE object_store.id = 2 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Out of range value for column 'allocated_size' at row 1
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)
	at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
	at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)

Therefore, this PR recalculates the allocated size in bytes for each Object Storage provider. It targets the 4.22.2.0 version. A follow-up PR targeting main will apply the same fix to environments already running 4.23.0.0.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

• Executed the normalization query locally and verified it recalculates correctly the allocated size for each Object Storage provider.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.32%. Comparing base (b56bbc2) to head (74017f5).

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14678   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16647    +3     
=========================================
  Files          5939     5939           
  Lines        537787   537787           
  Branches      65932    65932           
=========================================
+ Hits          98544    98553    +9     
+ Misses       428033   428023   -10     
- Partials      11210    11211    +1     

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.41% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vladimirpetrov]

@blueorangutan package

[blueorangutan]

@vladimirpetrov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20019

[vladimirpetrov]

LGTM based on manual testing. Tested with upgrades from 4.20 -> 4.22.1 -> PR and 4.20 -> PR.

[dheeraj12347]

@bernardodemarco
I reviewed the migration and tested the SQL locally with multiple active buckets. The allocated size was recalculated correctly in bytes, and the migration is picked up by the existing upgrade mechanism.

I also checked the failing Simulator CI job. The failure is in test_secondary_storage.py::test_01_sys_vm_start, which checks SSVM startup. I couldn't find a connection between that failure and the S3 bucket allocation change, although the underlying CI failure is still unresolved.

No issues found with the SQL change from my testing.