HBASE-30344 Wait for RegionServer sync replication state

[Sigma-Ma]

JIRA: https://issues.apache.org/jira/browse/HBASE-30344

What changes were proposed in this pull request?

Wait for the RegionServer serving the test table to observe the DOWNGRADE_ACTIVE sync replication state before verifying the replicated data in TestSyncReplicationStandbyKillMaster.

Why are the changes needed?

After Master recovery, the peer state returned by Admin can become DOWNGRADE_ACTIVE before the RegionServer refreshes its local peer state. The test may start reading while the RegionServer still treats the peer as STANDBY, causing DoNotRetryIOException.
The production-side STANDBY protection remains unchanged.

How was this patch tested?

Ran TestSyncReplicationStandbyKillMaster five consecutive times with Surefire retries disabled.

for run in 1 2 3 4 5; do
  mvn -pl hbase-server -am \
    -Dtest=TestSyncReplicationStandbyKillMaster \
    -Dsurefire.failIfNoSpecifiedTests=false \
    -Dsurefire.rerunFailingTestsCount=0 \
    -DskipITs \
    test || exit 1
done

[Apache9]

In general, if a peer is still under transiting, we should not get its final state?
Could you please explain more on the details?

Thanks.

[Sigma-Ma]

In general, if a peer is still under transiting, we should not get its final state? Could you please explain more on the details?

Thanks.

Thank you for pointing this out. You are right that, under normal circumstances, the transit RPC completes only after the procedure has finished.

The subtlety in this test is that the Master is intentionally aborted in preTransit, so the client may fail before receiving the procedure ID and cannot use the RPC completion as the signal. During recovery, TRANSIT_PEER_NEW_SYNC_REPLICATION_STATE updates the Master-side state to DOWNGRADE_ACTIVE before REFRESH_PEER_SYNC_REPLICATION_STATE_ON_RS_END refreshes the RegionServers. As a result, the getter can return DOWNGRADE_ACTIVE while a RegionServer still has STANDBY in its local cache.

The additional wait is intended to cover this window before verify() sends reads to the RegionServer. I hope this clarifies the reason for the change.