HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS - #8583
Open
nirdosh0110 wants to merge 2 commits into
Open
HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS#8583nirdosh0110 wants to merge 2 commits into
nirdosh0110 wants to merge 2 commits into
Conversation
2 tasks
apurtell
approved these changes
Aug 31, 2026
Apache9
requested changes
Sep 1, 2026
| cleanupMergedRegion(env); | ||
| break; | ||
| case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: | ||
| break; |
Contributor
There was a problem hiding this comment.
Why we need this change? In general, when rolling back a procedure, finally we will arrive the MERGE_TABLE_REGIONS_CLOSE_REGIONS state and reopen the parent regions?
nirdosh0110
force-pushed
the
HBASE-30334
branch
from
September 1, 2026 07:07
b9686fa to
cceda8e
Compare
added 2 commits
September 1, 2026 12:41
…CHECK_CLOSED_REGIONS MergeTableRegionsProcedure.rollbackState() previously had a bare `break;` for the MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS case. When rollback fires at this state (e.g. checkClosedRegions() throws on a stale recovered.edits file), the parents were expected to be reopened on the next rollback tick through MERGE_TABLE_REGIONS_CLOSE_REGIONS. In production this cascade did not fire reliably: an OpenTSDB tsdb merge observed on cluster hbase31a on 2026-08-05 stayed RIT for 48 minutes until operator intervention, with no TRSP submitted for either parent during rollback. Mirror the pattern SplitTableRegionProcedure.rollbackState() already uses for the symmetric SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS state: call the reopen helper (rollbackCloseRegionsForMerge) directly on the failing state's own rollback body, and make the follow-on MERGE_TABLE_REGIONS_CLOSE_REGIONS an explicit no-op with the mirror comment from SplitTableRegionProcedure so duplicate TransitRegionStateProcedures are not submitted for parents already OPENING. The invariant is that any pre-PONR state that can fail owns its own rollback reopen, rather than relying on the framework cascading to the next state.
…_REGIONS failure Plants a stale recovered.edits file on a parent so that MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact trigger observed on the hbase31a stuck-RIT incident on 2026-08-05), submits a merge, and asserts both parents transition back to OPEN after rollback. Verifies the rollback body reopens the parents on the failing state's own tick, mirroring the pattern SplitTableRegionProcedure already exercises for SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS.
nirdosh0110
force-pushed
the
HBASE-30334
branch
from
September 1, 2026 07:11
cceda8e to
ca58f15
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONScase inMergeTableRegionsProcedure.rollbackState()was a barebreak;. When rollback fires at this step (e.g.checkClosedRegionsfails), the parent regions stay CLOSED and produce an indefinite stuck RIT with no automatic recovery.CHECK_CLOSED_REGIONStoCLOSE_REGIONSso the existingrollbackCloseRegionsForMergehelper reopens the parents viaAssignmentManagerUtil.reopenRegionsForRollback.SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONShandling inSplitTableRegionProcedure.rollbackState()(which callsopenParentRegion(env)).JIRA: https://issues.apache.org/jira/browse/HBASE-30334
Test plan
TestMergeTableRegionsProcedurestill pass.checkClosedRegionsto throw in a merge procedure and confirm parents get reopened on rollback (no stuck RIT).