Repository navigation
HDDS-17240. Repair old volume quota without losing unflushed volume changes - #11846
Conversation
…hanges The repair found legacy volumes through the RocksDB iterator and put the row it read back into the fully cached volume table. A volume delete, a bucket count or tenant refCount change applied just before it but not yet flushed was rolled back, and a volume created but not yet flushed kept its old quota, so OMs diverged depending on how far their double buffer had flushed. Its response also wrote each volume under the bare name, so the conversion was lost on restart and the volume was listed twice. Both are fixed together: correcting only the key would make the rolled back value durable.
| for (OmVolumeArgs volArgs : volumeArgsMap.values()) { | ||
| metadataManager.getVolumeTable().putWithBatch(batchOp, volArgs.getVolume(), volArgs); | ||
| metadataManager.getVolumeTable().putWithBatch(batchOp, | ||
| metadataManager.getVolumeKey(volArgs.getVolume()), volArgs); |
There was a problem hiding this comment.
(non-blocking) Follow-up: the bare-name rows written by earlier repairs stay in RocksDB and are loaded into the volume table cache on restart. listAllVolumes iterates that cache, so an admin ozone sh volume list shows each repaired volume twice, and VolumeUtilizationMetrics emits two records for it. volume delete removes only the /vol key, so the bare row remains after the volume is deleted.
Could we file a follow-up JIRA to remove these rows (for example, a row whose key has no / prefix and equals its getVolume())?
There was a problem hiding this comment.
Thanks for the verifying! I will file a new Jira ticket for it.
|
Thanks @rjgoyln for the patch, @chungen0126, @smengcl for the review. For future PRs please add |
What changes are proposed in this pull request?
Full quota repair (
ozone repair om quota startwithout--buckets, or theQUOTAupgrade action) converts volumes with the pre-1.1 quota value-2.Previously, it read volumes from RocksDB and wrote them back to the fully cached volume table, potentially overwriting unflushed changes. This could restore deleted volumes, lose usage counter updates, overwrite recently updated quotas, or miss newly created volumes.
The response also wrote converted volumes under bare names instead of the proper volume table keys, causing the repair to be lost on restart and volumes to appear twice in listings.
This patch reads candidates from the volume table cache, rereads each volume under its lock before updating the quota, and writes the result using the proper volume table key.
Existing bare-name rows from previous repairs are not removed and may still appear in whole-table listings.
What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-17240
How was this patch tested?
testOldVolumeQuotaRepairKeepsUnflushedVolumeChangesto cover unflushed deletions, usage counter changes, quota updates, and volume creation. All cases fail on master.testQuotaRepairForOldVersionVolumeBucketto verify the persisted quota and ensure no bare-name row is written.TestOzoneRepairShellpass.Generated by Claude Code (Opus 5.5).