Skip to content

HDDS-17358. Closing the OM DB during a compaction DAG prune batch stalls and can crash the OM when the pruner reads the freed DB - #11705

Merged
jojochuang merged 2 commits into
apache:masterfrom
smengcl:HDDS-17358
Oct 30, 2026
Merged

jojochuang merged 2 commits into
apache:masterfrom
smengcl:HDDS-17358

Conversation

@smengcl

@smengcl smengcl commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

RocksDBCheckpointDiffer.close() held the differ lock (synchronized (this)) for the whole Scheduler.close() call, which interrupts the tasks and waits up to 60 seconds for them. A pruneSstFileValues task that was waiting for that lock between two compaction log entries cannot react to the interrupt, so every close during a prune batch stalled RDBStore.close() for 60 seconds. After the wait the task took the lock and kept reading the compaction log table while RDBStore flushed and freed the DB, which can crash the OM with SIGSEGV in RocksDB.get.

This PR:

  • Sets closed under the lock in close(), then shuts the scheduler down after releasing it, so a task waiting for the lock can get it and stop.
  • Makes every DB use by the scheduled tasks check closed first, under the same lock: pruneSstFileValues ends its batch, getOlderFileNodes returns nothing and removeKeyFromCompactionLogTable returns. No task touches the DB after close() returns, whatever the bounded scheduler wait returns. This is the same closed flag the compaction listeners already check.

Nothing changes when no prune task is running.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17358

How was this patch tested?

Two new tests in TestRocksDBCheckpointDiffer:

  • testPruneTasksDoNotUseDbAfterClose: running both prune tasks after close() leaves the compaction log entry in the table and in the prune queue. Without the fix the entry is deleted.
  • testCloseDoesNotWaitForTaskBlockedOnDifferLock: a scheduled task that tries to take the differ lock while close() waits for it does not keep close() waiting. Without the fix close() is still waiting after 10 seconds.

Ran locally: TestRocksDBCheckpointDiffer passed (58 tests, 0 failures, including both new tests). Checkstyle on rocksdb-checkpoint-differ reports 0 violations. CI will run on this PR.

🤖 Generated with Claude Code

…lls and can crash the OM when the pruner reads the freed DB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 22, 2026 22:33
@smengcl smengcl added the AI-gen label Oct 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@smengcl smengcl added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Oct 22, 2026

@SaketaChalamchala SaketaChalamchala left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @smengcl . lgtm. Raised this as a follow-up HDDS-17565

@smengcl
smengcl marked this pull request as ready for review October 30, 2026 01:48
@jojochuang
jojochuang merged commit c4b180d into apache:master Oct 30, 2026
134 checks passed
@jojochuang

Copy link
Copy Markdown
Contributor

Merged. Thanks @SaketaChalamchala @smengcl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-gen snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants