Skip to content

HBASE-30350 Fix stale Region-In-Transition tracker entries on region deletion - #8594

Open
mnpoonia wants to merge 1 commit into
apache:masterfrom
mnpoonia:HBASE-GCRegionProcedure-rit-leak
Open

HBASE-30350 Fix stale Region-In-Transition tracker entries on region deletion#8594
mnpoonia wants to merge 1 commit into
apache:masterfrom
mnpoonia:HBASE-GCRegionProcedure-rit-leak

Conversation

@mnpoonia

@mnpoonia mnpoonia commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

RegionInTransitionTracker and RegionStates are two separate in-memory maps on the master that are not kept in sync automatically. Several code paths deleted a region from RegionStates directly without notifying the tracker, so a deleted region could be left behind in the tracker forever — surfacing as a permanently stuck "STUCK Region-In-Transition" warning that only clears on a full master restart (loadMeta() rebuilds the tracker from hbase:meta from scratch; a GC'd region isn't in meta, so no stale entry survives that path).

This was the root cause of a 9.8h stuck RIT incident: GCRegionProcedure deleted a no-longer-needed region from RegionStates but never told the tracker, leaving a stale entry that nothing but a restart could clear.

Fix: add AssignmentManager#deleteRegion(RegionInfo) and its plural form #deleteRegions(List<RegionInfo>) as the single sync point that updates both maps together, and route every region-deletion call site through it instead of calling RegionStates#deleteRegion(s) directly.

Confirmed live bug, reproduced end-to-end: GCRegionProcedure. Temporarily reverting its call site back to a direct RegionStates#deleteRegion() call reproduces the original incident symptom in testGCRegionProcedureRemovesStaleRegionInTransitionEntry (the tracker entry survives GC); restoring the fix makes the same test pass again.

Plausible but unconfirmed live bug: AssignmentManager#deleteTable. DeleteTableProcedure waits for RegionStates to clear RIT before calling it, but that wait only checks RegionStates, not the tracker — so it doesn't rule out a stale tracker entry, though it isn't proven reachable either.

Defensive hardening, not confirmed reachable today (the region being deleted at these sites is already untracked — either terminal/OFFLINE on a disabled table, or a non-default replica, which the tracker never tracks in the first place):

  • AssignmentManagerUtil#removeNonDefaultReplicas (split/merge)
  • EnableTableProcedure (replica-count-decrease cleanup)
  • RestoreSnapshotProcedure#deleteRegionsFromInMemoryStates

Test plan

  • testDeleteRegionRemovesStaleRegionInTransitionEntry / testDeleteRegionsRemovesStaleRegionInTransitionEntries — cover the singular/plural AssignmentManager#deleteRegion(s) sync point directly.
  • testGCRegionProcedureRemovesStaleRegionInTransitionEntry — runs the real GCRegionProcedure end-to-end through the procedure executor. Verified this test fails against the pre-fix code path and passes against the fix.
  • Ran TestAssignmentManager, TestAssignmentManagerUtil, TestEnableTableProcedure, TestRestoreSnapshotProcedure, and related suites — all green, no regressions.

@mnpoonia
mnpoonia force-pushed the HBASE-GCRegionProcedure-rit-leak branch from e6403df to a29d901 Compare August 31, 2026 18:07
…deletion

RegionInTransitionTracker and RegionStates are two separate in-memory
maps on the master that are not kept in sync with each other
automatically. Several code paths deleted a region from RegionStates
directly without notifying the tracker, so a deleted region could be
left behind in the tracker forever -- surfacing as a permanently stuck
"STUCK Region-In-Transition" warning that only clears on a full master
restart (loadMeta() rebuilds the tracker from hbase:meta from scratch;
a GC'd region isn't in meta, so no stale entry survives that path).

This was the root cause of a 9.8h stuck RIT incident: GCRegionProcedure
deleted a no-longer-needed region from RegionStates but never told the
tracker, leaving a stale entry that nothing but a restart could clear.

Fix: add AssignmentManager#deleteRegion(RegionInfo) and its plural
form #deleteRegions(List<RegionInfo>) as the single sync point that
updates both maps together, and route every region-deletion call site
through it instead of calling RegionStates#deleteRegion(s) directly.

GCRegionProcedure is a confirmed live bug: reverting its call site
back to a direct RegionStates#deleteRegion() call reproduces the
original incident symptom in
testGCRegionProcedureRemovesStaleRegionInTransitionEntry (the tracker
entry survives GC), and restoring the fix makes the same test pass
again.

AssignmentManager#deleteTable is a plausible but unconfirmed live bug
of the same shape: DeleteTableProcedure waits for RegionStates to
clear RIT before calling it, but that wait only checks RegionStates,
not the tracker, so it does not rule out a stale tracker entry.

Three additional call sites with the same code pattern were audited
and fixed defensively, even though the region being deleted there is
already untracked (either already terminal/OFFLINE on a disabled
table, or a non-default replica, which the tracker never tracks in
the first place) -- so these are hardening against future regressions
rather than fixes for currently-reachable bugs:
  - AssignmentManagerUtil#removeNonDefaultReplicas (split/merge)
  - EnableTableProcedure (replica-count-decrease cleanup)
  - RestoreSnapshotProcedure#deleteRegionsFromInMemoryStates

Includes regression tests covering the singular and plural delete
paths, plus an end-to-end test that runs the real GCRegionProcedure
through the procedure executor.
@mnpoonia
mnpoonia force-pushed the HBASE-GCRegionProcedure-rit-leak branch from a29d901 to ca73a9a Compare August 31, 2026 18:28
@mnpoonia

Copy link
Copy Markdown
Contributor Author

Looking for approvals for running the github actions workflow
@Apache9 @virajjasani

@Umeshkumar9414

Copy link
Copy Markdown
Contributor

We only delete an region once it is closed. I can think of 3 scenario of region close- 1. Table disabled 2. Merge parents 3. Split parent . In all 3 cases, regions should be out of Region-In-trnasition tracker when region was getting closed so we should never need to handle it while deleteing.

I think bug here, that we need to find out, is why it was not removed from tracker while closing. I am opposed to the change in tracker when actually there is no state change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants