HDDS-16640. Make KeyLifecycleService configuration reconfigurable at runtime by priyeshkaratha · Pull Request #11721 · apache/ozone

priyeshkaratha · 2026-10-23T17:56:33Z

What changes were proposed in this pull request?

Before this change, changing any ozone.om.key.lifecycle.service.* property meant restarting OM. This PR makes those properties reconfigurable at runtime through the OM reconfiguration framework (ozone admin reconfig --service=OM ... start).

Reconfigurable properties (the new KeyLifecycleService.OZONE_KEY_LIFECYCLE_PROPERTIES set):

ozone.lifecycle.service.enabled
ozone.lifecycle.service.interval
ozone.lifecycle.service.timeout
ozone.lifecycle.service.workers
ozone.lifecycle.service.delete.batch-size
ozone.lifecycle.service.mpu.abort.limit.per.task
ozone.lifecycle.service.delete.cached.directory.max-coun
ozone.lifecycle.service.state.save.interval.ms
ozone.lifecycle.service.state.save.keys.processed
ozone.lifecycle.service.move.to.trash.enabled

What is the link to the Apache JIRA

HDDS-16640

How was this patch tested?

Added test case and tested manually

bash-5.1$ vi /etc/hadoop/ozone-site.xml
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin om lifecycle status --service-host om
========================================
          Lifecycle Service Status
========================================
IsEnabled: true
IsSuspended: false
No buckets are currently being processed.
========================================
bash-5.1$ vi /etc/hadoop/ozone-site.xml
bash-5.1$
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin om lifecycle status --service-host om
========================================
          Lifecycle Service Status
========================================
IsEnabled: false
No buckets are currently being processed.
========================================

…runtime

adoroszlai · 2026-10-23T18:24:01Z

Thanks for the patch. To save CI resources, please consider creating PRs as draft, and keeping them that way until approved. (No need to change this now, just a note for future PRs.)

ChenSammi · 2026-10-26T07:35:17Z

Reconfigurable properties (the new KeyLifecycleService.OZONE_KEY_LIFECYCLE_PROPERTIES set):

ozone.om.key.lifecycle.service.enabled
ozone.om.key.lifecycle.service.interval
ozone.om.key.lifecycle.service.timeout
ozone.om.key.lifecycle.service.workers
ozone.om.key.lifecycle.service.delete.batch-size
ozone.om.key.lifecycle.service.mpu.abort.limit.per.task
ozone.om.key.lifecycle.service.delete.cached.directory.max.count
ozone.om.key.lifecycle.service.state.save.interval.ms
ozone.om.key.lifecycle.service.state.save.keys.processed
ozone.om.key.lifecycle.service.move.to.trash.enabled

These property names in the description not match the real name

Will the "ozone admin reconfig --service=OM --address=om:9862 start" command display which property name is changed?

@priyeshkaratha , could you add a section about reconfiguration in lifecycle.md and it's Chinese version?

adoroszlai · 2026-10-26T07:38:26Z

could you add a section about reconfiguration in lifecycle.md and it's Chinese version?

Please add documentation in ozone-site. Leftover docs are being removed from this repo in #10829.

ChenSammi · 2026-10-26T07:39:39Z

+  public void reconfigure(String property, ConfigurationSource conf) {
+    switch (property) {
+    case OZONE_KEY_LIFECYCLE_SERVICE_ENABLED:
+      isServiceEnabled.set(conf.getBoolean(property, OZONE_KEY_LIFECYCLE_SERVICE_ENABLED_DEFAULT));


I believe we should call the start() if the change is from false to enabled, and do nothing if it's from true-> false.

ChenSammi · 2026-10-26T07:41:38Z

+  /**
+   * Apply the new value of a property in {@link #OZONE_KEY_LIFECYCLE_PROPERTIES
+   * } without restarting OM.
+   * Changes to batch sizes and limits take effect on the next scheduled task. A new interval restarts the service,


"Changes to batch sizes and limits take effect on the next scheduled task" is not accurate, all the changes, except the interval, will take effect immediately on existing tasks' next batch scan.

ChenSammi · 2026-10-26T07:44:12Z

+   * @param property the property being reconfigured
+   * @param conf configuration holding the new value, or not holding the property to fall back to the default
+   */
+  public void reconfigure(String property, ConfigurationSource conf) {


Instead of for each property change to take effect individually, is it possible that we can handle all the property change in one function in the same time?

Yes, it is handled

ChenSammi · 2026-10-26T07:46:36Z

+    getSubject().reconfigureProperty(OZONE_KEY_LIFECYCLE_SERVICE_ENABLED, String.valueOf(originalEnabled));
+    getSubject().reconfigureProperty(OZONE_KEY_LIFECYCLE_SERVICE_INTERVAL, originalInterval);
+    getSubject().reconfigureProperty(OZONE_KEY_LIFECYCLE_SERVICE_WORKERS, originalWorkers);
+    assertEquals(originalEnabled, service.status().getIsEnabled());


Could you also add test for unset case?

priyeshkaratha · 2026-10-27T05:12:04Z

Hi @ChenSammi Addressed review comments in latest commit. Created https://issues.apache.org/jira/browse/HDDS-17499 for document changes.

ChenSammi · 2026-10-28T03:06:41Z

Thanks @priyeshkaratha for updating the patch.

One minor comment is we'd better change the log message from "KeyLifecycleService is suspended, disabled, or leader not ready" to ""KeyLifecycleService is suspended, disabled, leader not ready, or about to restart".

And the introduced "OZONE_KEY_LIFECYCLE_PROPERTIES" reminds me that maybe we should extract all the key lifecycle properties from OMConfigKeys, to a new file, similar to "OmConfig", and extend the ReconfigurableConfig, that will make the logic very clear, and also easy to maintain. What do you think?

priyeshkaratha · 2026-10-28T04:17:37Z

And the introduced "OZONE_KEY_LIFECYCLE_PROPERTIES" reminds me that maybe we should extract all the key lifecycle properties from OMConfigKeys, to a new file, similar to "OmConfig", and extend the ReconfigurableConfig, that will make the logic very clear, and also easy to maintain. What do you think?

Thanks @ChenSammi. It makes sense to me too. It may require more changes. I have created followup Jira HDDS-17515 for the same

priyeshkaratha · 2026-10-29T06:41:05Z

Thanks @ChenSammi for the review

HDDS-16640. Make KeyLifecycleService configuration reconfigurable at …

a49da84

…runtime

priyeshkaratha requested a review from ChenSammi October 23, 2026 17:56

github-actions Bot added the om label Oct 23, 2026

ChenSammi reviewed Oct 26, 2026

View reviewed changes

adoroszlai marked this pull request as draft October 26, 2026 08:01

addressing review comments

84f3be9

priyeshkaratha requested a review from ChenSammi October 27, 2026 05:12

changing log message

ae3ba22

ChenSammi approved these changes Oct 29, 2026

View reviewed changes

priyeshkaratha marked this pull request as ready for review October 29, 2026 05:11

priyeshkaratha merged commit 91ca736 into apache:master Oct 29, 2026
133 checks passed

Conversation

priyeshkaratha commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai commented Oct 23, 2026

Uh oh!

ChenSammi commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai commented Oct 26, 2026

Uh oh!

ChenSammi Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

ChenSammi Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

ChenSammi Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha commented Oct 27, 2026

Uh oh!

ChenSammi commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

priyeshkaratha commented Oct 28, 2026

Uh oh!

Uh oh!

priyeshkaratha commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

priyeshkaratha commented Oct 23, 2026 •

edited

Loading

ChenSammi commented Oct 26, 2026 •

edited

Loading

ChenSammi Oct 26, 2026 •

edited

Loading

ChenSammi commented Oct 28, 2026 •

edited

Loading