Skip to content

fix: call replication/failback on target cluster/pool/volume resolved… - #472

Open
geoffrey1330 wants to merge 16 commits into
mainfrom
refactor_snapshot_replication-failback
Open

fix: call replication/failback on target cluster/pool/volume resolved…#472
geoffrey1330 wants to merge 16 commits into
mainfrom
refactor_snapshot_replication-failback

Conversation

@geoffrey1330

@geoffrey1330 geoffrey1330 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

… via relationship lookup

fix: failback correctness — slot flip-flop, DeleteSource, and test coverage

Problem

After a failover, triggering a failback left volumes stuck or oscillating:

  1. Wrong cluster targeted — the ops controller called the failback API on the source cluster instead of resolving the relationship to find the target cluster where the failed-over volume actually lives.
  2. Slot stuck in FailedOverreconcileSyncStatus had no handler for cutover_done/failed_over with is_source: false, so the slot never transitioned back to replicating/source.
  3. Slot flip-flopreconcileReplicating unconditionally set cutover_done/target when it saw a done state, racing against reconcileSyncStatus and causing infinite oscillation between the two states.
  4. do_replicate not restored — the original source lvol had do_replicate cleared during failover; _swap_failback_lvol_uuid blindly copied that False value, so replication never resumed after failback.

Solution

  • Route failback API calls via relationship lookup to the correct cluster/pool/volume.
  • Add is_source: false detection in both reconcileSyncStatus and reconcileReplicating — when the backend reports a terminal cutover state with is_source: false, both paths now agree to write replicating/source, stopping the oscillation.
  • Drive failback slot state via slot controller polling instead of premature ops controller patches.
  • Extend DeleteSource to cover failback: deletes the failed-over volume on the target cluster when set (previously only worked for migration).
  • Restore do_replicate = true and replication cadence config from the failback source after UUID swap.
  • Extract backendStateFailedOver constant (goconst lint).

Tests

  • Fixed TestSlot_ReconcileReplicating_FailedOver_UpdatesState and TestSlot_ReconcileReplicating_DetectsCutoverDone_SetsDirectionTarget — both had IsSource: false but were testing the forward failover path; corrected to IsSource: true.
  • Added TestSlot_ReconcileReplicating_CutoverDone_Failback_SetsReplicatingSource and TestSlot_ReconcileReplicating_FailedOver_Failback_SetsReplicatingSource covering the is_source: falsereplicating/source path.
  • Updated ops controller tests to mock relationship lookup.

Test plan

  • go test ./internal/controller/... green
  • deleteSource: true removes the failed-over target volume (targetLvolID changes post-failback)
  • No flip-flop in kubectl get replicationslot -w

…er states with is_source check in reconcileReplicating
…get cluster where cutover_pending record lives
…ath, fast-requeue stale case, and persist annotation deletions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant