fix: preserve original source lvolID through failback cutover by swap… - #1276
Open
geoffrey1330 wants to merge 131 commits into
Open
fix: preserve original source lvolID through failback cutover by swap…#1276geoffrey1330 wants to merge 131 commits into
geoffrey1330 wants to merge 131 commits into
Conversation
…turns a dict rpc_client.subsystem_get returns ONE subsystem dict (single_or_none), not a list. Indexing it with [0] raised KeyError(0), the helper's best-effort except swallowed it as a warning, and the eviction silently never executed — run 20260824_104449 failed with the same 40x add_ns -32602 while the fix was nominally in place. The test fake modelled the wrong shape too (a list), which is exactly how the bug got past the suite; it now models reality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ver adds nvmf_subsystem_remove_ns acknowledges before it completes (the same async false-success that dropped a shared subsystem in the PVC-expand incident, whose fix polls for confirmation). The fail-back eviction removed the stale namespace and add_ns raced the removal and lost on all 8 retries -- run 20260824_110959: 40 evictions logged, 40 add_ns -32602 right behind them. The eviction now polls the subsystem until the namespace is actually gone (bounded, 20s) before returning, matches the stale entry by uuid as well as nsid (the preserved identity collides on both axes), and the test fake now models the acknowledged-but-lingering removal so this race stays pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ot just the primary Run 20260824_113711 proved the eviction itself now works: the primary's add_ns returned result:1 for the first time. But the preserved-NQN subsystem exists on EVERY node of the recovered HA set, each still holding the original volume's namespace at the preserved nsid. The HA peer's add_ns failed with the same -32602 the eviction was written for, add_lvol_on_node's peer failure rolled the whole cutover back, and all 5 fail-back cutovers died on max retry (0/5). _create_target_lvol_clone now calls _evict_stale_namespace for each online HA peer right before that peer's add_lvol_on_node, exactly as it already did for the primary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…attempt; add soak cases 7-9 Case 6 (run 20260824_144226) exposed the stall: a node outage mid-create left a REP_* landing volume whose id was never stored on the task, and every retry of the transfer died on "LVol name must be unique" (~31s loop for the rest of the run), stalling three volumes' chains and the case behind the 180s lag gate. The runner now probes for a record already wearing the derived name before creating: adopt it when online, wait when in_deletion, force-delete when half-created. New soak cases per the extended test plan: - case 7: 20 namespaced volumes on 2 shared subsystems (10 ns each), randomly assigned across 2 clients, replication + fail-over + fail-back with NQN/nsid identity asserted per namespace. The deployer gains an add_client mode to grow an existing lab to 2 clients. - case 8: sequential-pressure catch-up: 64k/QD64/4-job fills of 50G per volume, repeated; peak backlog and catch-up time are recorded and the backlog must drain under the lag gate every cycle. - case 9: chaos: random SPDK-container kills on BOTH clusters' nodes while replication runs (seeded, logged with the pipeline phases active at each kill), then full catch-up + fail-over integrity verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sn shutdown's stderr Run 20260824_153107: case 6's sn shutdown was silently REFUSED because the source cluster was still ACTIVE - REBALANCING from the previous case's restore (open device_migration + balancing_on_restart tasks); the refusal went to stderr, which the driver's exec channel drops, so the test stared at an online node for the full 900s budget. Cases 5 and 6 now wait for the victim's cluster to settle before shutting the node down, and the shutdown command merges stderr so a refusal is visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… not nqn alone Soak case 7 (run 20260824_174611) caught this on its first run: namespaced volumes SHARE a subsystem, so replicate_lvol_on_target_cluster's existing-copy guard -- an nqn-only match -- fired for every namespace after the first. Namespace 1's fail-over copy already carried the shared nqn, so namespaces 2..N returned ITS target lvol id and were never failed over at all: 9 of 10 volumes silently absent after a DR fail-over, with every call reporting success. The guard now compares nqn AND ns_id, which is the identity a fail-over copy actually preserves. The existing idempotency test passed only because its fake left the already-failed-over copy at the model-default nsid while the source carries nsid 7; a real copy preserves the source's nsid. Fixed, and a companion test pins that a SIBLING namespace on the same nqn is not mistaken for this volume. Also scale the leftover-volume drain budget with the number of victims: case 7 leaves 20 namespaced volumes plus their REP_* landing copies, and 37 of them did not drain inside the flat 300s, failing case 8 in its prologue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oss chaos kills CHAOS_EVENTS and friends can now be set per run (stage_and_run_repl_cases case9 CHAOS_EVENTS=100) instead of editing the driver on the box. Case 9 also records whether fio survived each kill: that is the promotion-window signal the spdk ANA-transition fix targets, so a chaos soak now reports it directly instead of only asserting recovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…face fail-over errors - SPDK_IMAGE -> main-2a03661a-amd64 (sha256:961410ae...), the first ultra build containing the promotion-window ANA-transition fix (spdk R26.3 554c80f11). Verified it was built FROM spdk-core:R26.3-latest whose manifest was created at 18:41:57, before this ultra build started at 18:42:52 -- the floating-tag manifest race makes that check necessary. - CLIENT_COUNT 2, so a namespaced (case 7) run does not need add_client. - do_failover now captures the controller's error and log output. A bare 'returned no connection strings' cost two lab runs to diagnose; the reason was being written to a stderr nobody read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two unrelated, pre-existing issues currently failing every PR's "Python checks" job (found while chasing an unrelated PR's CI failure — these files aren't touched by that PR at all, so the merge-commit CI check was just surfacing main's own already-broken state): - tests/unit/test_ucs_debounce_bound.py: unused `MagicMock` import (F401). - simplyblock_core/test/test_replication_chain_completeness.py: unused `import copy` (F401), and a lambda that sequenced `added.append(...)` with a return value via a `(side_effect, value)[-1]` tuple-index trick -- mypy's func-returns-value check flags embedding append()'s None result in the tuple literal. Replaced the lambda with a small nested function that does the append as its own statement and returns the real value normally; same behavior, no trick, no warning. No functional change -- verified via pytest (22 passed) and unittest (7 passed) on the two affected test modules.
Soak case 7 (run 20260824_215758) got 14 of 20 namespaces failed over and then died in add_ns on the 15th. The replication destination is chosen per volume by capacity (_get_next_3_nodes), with nothing tying volumes that SHARE a subsystem to one target node. Because a fail-over copy preserves the volume's NQN and nsid, scattering siblings splits a single shared subsystem across unrelated target primaries: each one advertises the same NQN carrying only its own subset of namespaces, and a sibling whose nsid is already taken there cannot be added. A volume that shares its NQN with an already-replicating sibling now inherits that sibling's replication node. Also raise the soak's fail-over diagnostic capture to DEBUG: the SPDK response behind 'Failed to add bdev to subsystem' is logged there, and the controller only re-reports its own generic message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a kill Run 20260824_224909 showed the soak damaging the cluster instead of testing it: the recovery gate accepted status == online while health_check was still False and the source cluster sat in SUSPENDED/IN_ACTIVATION, so kills kept landing on a half-recovered 2-node cluster. It now requires every node online AND healthy and every cluster active/degraded before the next event. fio also died at event 3 (the promotion-window EIO) and stayed dead, so the remaining 97 events would have run against an idle client. The case now reconnects and restarts the workload after an outage, and still reports every death. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gracefully shutting one node of a 2-node cluster made the SURVIVING node
abort its own SPDK application:
JC detected a network outage nd=1 njms=2
JC aborts the node due to network outage
spdk_abort_node: Forcing application shutdown via abort. (core dumped)
The journal component requires jc_ha_nmin_jms() reachable journals --
2 normally, 1 when the dual-node flag is set. A 2-node cluster losing its
peer is left with 1 of 2, so the survivor fail-stopped. Every client path
vanished at once, fio took hard EIO and XFS shut down (soak case 6), and
the cluster went SUSPENDED. The spdk fork implements the tolerance and
exposes the jc_set_dual_node RPC, documenting that in a dual-node
configuration a single connected JM is enough -- but nothing in the
control plane ever called it, so no 2-node cluster has ever had it on.
apply_jc_dual_node() now sets the flag across the cluster whenever a node
is added (so growing 2 -> 3 also CLEARS it) and whenever a node is
brought back up (a restarted node returns with the JC default). The flag
tracks MEMBERSHIP, not the online count: a 3-node cluster with one node
down must keep requiring two journals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…554c80f11) Iteration 12 of the 2026-08-24 run failed fio verify on four blocks, and the received magics were random (c32e, b24e, e6b4, c8b7) rather than fio's constant 0xacca. Stale-but-valid data would still carry acca and fail as an offset or crc mismatch, so those buffers never held an fio header at all -- the map named a location and the bytes handed back were not that location's content. The placement history exonerates the map at every observable point: no steady-state re-home across the nine dump points of iterations 11-12, every group back on its pre-outage home, and the three roles identical at range level (12,810 groups) at post_incident. ultra b44de698 fits that shape: a reactively promoted distrib used to signal JC leadership immediately while, under write protection, the parity desynchronisation check was still outstanding, so reads served in that window came off desynchronised parity. The signal now waits for the check to finish. spdk moves a311a6852 -> 554c80f11, which keeps the upstream retry-state fix (#3686) as an ancestor and adds the promotion-window ANA transition fix. The gate now pins the ultra commit as well, so a stale image cannot quietly re-test the build that already failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a retained count With the JC dual-node fix in place case 6 finally kept the client alive through a primary outage (fio_alive=True, replicated 5 -> 13 during the outage) and then failed on its own assertion: during=13, after=10. replicated_count counts RETAINED replicated snapshots, and retention keeps only the newest generations, so it is bounded and falls after a burst. It cannot express 'replication resumed'. Cases 5 and 6 now require a point-in-time created AFTER recovery to reach the target, which is what resumption actually means, using the harness primitive that already exists for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replication kept a flat count of internal snapshots, so history never went
back further than a couple of cadence ticks and a fail-over could only ever
land on 'a minute ago' -- useless against a logical corruption, which the
newest copy has faithfully replicated.
A replication policy can now carry a retention SCHEDULE:
--retention-schedule '15m:2h,1h:11h,1d:7d'
'one snapshot every 15 minutes for the last 2 hours, then hourly for 11
hours, then daily for 7 days'. Snapshots past the total span are pruned.
Selection is a pure function of (times, schedule, now) in the new
snapshot_retention module, so it is unit-tested without a cluster: a
minute-cadence stream over 8 days collapses to ~25 retained snapshots that
still reach back 6+ days.
The schedule never overrides MIN_KEEP_REPLICATED: the newest pair is always
retained, because deleting a snapshot swap-merges its segments into the
successor chained to it. An unparseable schedule is rejected when the policy
is created, and if one ever reaches the runner it is logged and treated as
'no schedule' rather than crashing replication or silently dropping history.
Fail-over gains a generation selector: generation 0 is the newest replicated
point-in-time (previous behaviour), higher values walk back through the
retained history. Asking for more generations than exist is an explicit
error rather than a silent fall back to the newest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a supported configuration The product minimum is 3 nodes. The 2-node src/fresh clusters this lab used are exactly what produced the 2026-08-24/25 failure chain: with one node down the survivor holds 1 of 2 journal members and the JC aborts it (whole-cluster outage on a single node stop), and the restart rebalance has no third failure domain to place into, so device_migration loops on 'no allowed placement' forever and pins the cluster in REBALANCING, blocking sn shutdown and every node-down test behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A policy created with --consistency-group makes all attached volumes ONE crash-consistent unit: 1) Same-LVS invariant: the group pins to the first member's node/LVS. Attaching the policy to a volume elsewhere FAILS before any state is written; creating a volume under the policy forces placement onto the pinned node (an explicit conflicting --host is an error). 2) Group snapshots: an auto-managed ConsistencyGroup record lives and dies with the policy. One SPDK call (bdev_lvol_snapshot_group, spdk branch consistency-groups) freezes IO on every member blob, snapshots them one after the other, and unfreezes; SPDK unfreezes-first-then-GCs on mid-sequence failure. The controller mirrors the single-snapshot path per member around that call: replica registration, records, chain links, events, replication tasks. All-or-nothing: any registration or record failure rolls the whole generation back, and the group_seq counter moves only after full success. 3) The cadence (snapshot_monitor) snapshots CG policies as a group - one generation per tick with group-wide back-pressure - and its members leave the per-volume loop. 4) Membership epochs: a late joiner is active from the FIRST group snapshot after its attach (joined_seq = last_group_seq + 1); a detached member's epoch closes at the current generation. Failing over to generation N warns which current members that point-in-time does NOT contain and which contained volumes are no longer members - in the controller result, the CLI failover table, and the API response (200 + warnings body instead of the empty 204). Group provenance (group_id, group_seq) is stamped on member snapshots and travels onto the replicated target copies, which is what the fail-over generation selector actually returns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… its namespace bdev_lvol_clone_register acknowledges before the bdev is examinable -- the third member of the acknowledge-before-complete family, after remove_ns (PVC-expand, eb127ee) and the case-3 eviction. The HA-peer leg of a namespaced fail-over issues nvmf_subsystem_add_ns immediately after the register and lost the race every time: -32602 with the peer's subsystem EMPTY, while the bdev existed moments later (run 20260825_122423, LVS_13/LVOL_121, uuid f01ea33c). The peer failure then rolled back the whole fail-over of that namespace. The stack build now polls get_bdevs (bounded, 20s) after clone_register and only then proceeds to the namespace add. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…inery
A finished transfer retires its REP_ landing volume by emptying the
record's bdev_stack (the blob lives on as the converted, chained
snapshot) and removing the record. Routing that retirement through
delete_lvol flipped the record to in_deletion first, so any interruption
before remove() stranded a record the monitor can never finish: with an
empty stack there is nothing to issue, the delete-status poll answers 4
('no async delete request exists') forever -- 856x in 30 minutes in run
20260825_125156 -- and every cleanup that waits for volumes to drain
times out behind it (case 7: 33 stuck, case 9: 2 stuck, both dead in
their prologues).
Two-sided fix:
- the retirement path tears down the nvmf plumbing DIRECTLY per node
(delete_lvol_from_node(force=True); the empty stack means no blob
work) and then removes the record, never entering in_deletion;
- the monitor retires any in_deletion record with an empty bdev_stack
record-only. Deliberately NO fallback delete of top_bdev: that bdev IS
the converted snapshot, deleting it would destroy replicated data.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
app_LVolMonitor received only the SHARED files, so a fix living in lvol_monitor.py itself (the empty-stack record retirement) verified green while the running monitor never contained it -- the import probe checks the shared modules, not the service's own file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
case 10 -- online migration under sustained heavy IO: 3 volumes per source node at 8k randwrite QD64 x 4 jobs, replication must reach and hold a bounded lag against that load, the final online migration must complete for every volume with fio uninterrupted, and the IO freeze during each cutover is MEASURED as the client actually experiences it (100ms heartbeat probe per volume; the largest gap inside the commit window is the freeze; 30s sanity bound). case 11 -- retention ladder + generation fail-overs: policy with --retention-schedule 5m:15m,7m:30m,10m:1h, ~2h of history under a 30s-fsynced record stream, then (a)/(b) kept-and-pruned verified INDEPENDENTLY of the retention code (gap/count/horizon analysis per tier), and (c) three rounds of fail-over to a randomly selected older generation with exact-data validation (every record up to the generation's snapshot present, none after), each followed by a full fail-back cutover. case 12 -- consistency groups (gated: requires the consistency-groups build; refuses with a clear message otherwise): CG policy, 3 members pinned to one LVS, strictly ordered fsynced writes A->B->C, group generations verified complete (every group_seq covers all members), retention per schedule, then fail-over to the latest and to a random earlier generation verifying BOTH generation correlation and crash consistency: seq(A) >= seq(B) >= seq(C), max skew 1. Policies with extra flags get distinct names so a schedule/CG policy never silently reuses a plain one another case left behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts NQN; pin the master-line image _remove_lvol_subsys_from_node deleted the subsystem whenever it observed it empty. For per-volume subsystems that is correct cleanup; for NAMESPACED volumes the shared subsystem is legitimately empty in the window between one member's teardown and the next member's add -- and a stuck in_deletion member's retry loop observes that window sooner or later. Run 20260825_224221: 8 of 20 namespaced fail-overs landed, then a looping rollback record deleted the shared subsystem on the HA peer, and every following member's nvmf_subsystem_add_ns died -32602 against a missing subsystem. Delete-on-empty now checks for other live claimants of the NQN on the node first; the LAST member out still removes it. Also pin the deployer to sha256:a3854cd4 (main-d91ff03a-amd64), the first ultra build FROM spdk-core:master-latest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the outage
Two defects found by the 2026-08-25 soak, both in the repair path.
Duplicate paths. repair_multipath_controller() is fanned out over a thread
pool with no serialization, so two workers read the same missing={ip} and both
attach it; SPDK admits both because its -EEXIST guard runs before the async
probe and compares only the active path, and the target issues two cntlids. The
result on 2026-08-25 was remote_jm_1e7ff71e carrying (96.179, 97.9, 97.9). It
was never repaired because _collect_attached_ips() returns a SET, so the
duplicate read as 2-of-2 and the control plane called the node healthy while
the soak's own verifier counted 3-of-2 for 900 s and gave up.
So: a per-(node, controller) lock, non-blocking because the loser of the race
has nothing to add; duplicate_attached_paths() over a path LIST; a prune that
detaches the duplicated address and lets the existing missing-path loop
re-attach it exactly once (SPDK's bdev_nvme_delete removes every controller
matching a trid, so one copy cannot be singled out); and a refusal to prune
when the address is the only one attached, which would take the bdev down
instead of repairing it. health_controller now reports a duplicate as
UNHEALTHY rather than passing it silently.
Dial holds outliving their cause. dial_backoff exists so a refusing address
cannot burn a healthy node's app thread on connect polling, and that is still
wanted. But the ceiling was 300 s and a hold clears only on a success that
allowed() refuses to let anybody attempt, so it had to time out. Every
all-nodes NIC flap in run 20260825_155730 therefore stalled path healing on a
plateau of exactly 15 missing paths -- all of them one held address -- for
250-306 s, i.e. the ceiling, while the address had been reachable within 30 s.
All three data corruptions in that run happened inside those windows.
Ceiling drops to 60 s: a probe once a minute does not burn app-thread time, so
the breaker keeps its purpose while bounding how long a returning path stays
unrepaired. New clear() drops a hold on evidence of reachability, wired to the
one piece of evidence available locally -- an address with a live enabled path
on this very controller. Deliberately NOT driven by the peer's DB status: a
node whose record says ONLINE while its SPDK is dead is the case this module
was written for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…longer restart wait The harness ran for 1h56m through a data corruption and reported PASS on every checkpoint. fio prints verify failures WITHOUT the "fio: " prefix -- verify: bad magic header a8a4, wanted acca at file .../vol2/soak_mp_2.1.0 -- and FIO_HARD_ERROR_MARKERS carried "fio: verify" and "verify failed", neither of which matches that. So vol6 (18:53) and vol2 (19:43) corrupted silently while their fio kept running, and the fault surfaced only at 20:49 when vol4's fio *process* died and the rc-file branch caught it. A data verification failure is the most important thing this harness can find; it must never again depend on fio also crashing. Markers now cover the unprefixed forms, checked against the three real failure lines from that run. verify_dump=1 so a mismatch writes the received and expected 4 KiB buffers. Every corruption so far has died with the returned bytes unidentified -- we could not distinguish stale data from parity noise from a neighbouring block, and the volumes live on instance store, so they vanish when the fleet stops. That is how the 08-24 evidence was lost, and it nearly repeated on 08-25. RESTART_TIMEOUT passthrough in the launcher: the 900 s default aborted run 20260825_085018 at iteration 4, where a JC abort plus a stranded controller reset meant both nodes needed ~36 min to return -- and they did return healthy, with no fio error. Until that recovery time is fixed, a longer wait measures the product rather than the harness's patience. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dirty bitmap has been in the SPDK fork for a long time and has never executed. bdev_lvol_transfer takes an optional allow_partial; the control plane never sent it, and could not: a partial transfer ships only the ranges written since the previous snapshot, so the destination must already hold everything else -- and this pipeline created a FRESH EMPTY landing volume, transferred the whole snapshot into it, and only chained it afterwards. A delta into an empty volume silently drops every cluster it does not cover. Chain the landing volume onto the destination's copy of the previous snapshot BEFORE the transfer instead of after it. That is the same bdev_lvol_add_clone the finish step already issued, moved earlier, and it is what makes the volume a valid delta target: a cluster the new snapshot does not own stays unallocated and reads through to the parent, and the first write into a cluster the delta does touch makes the blobstore copy-on-write the whole cluster from that same parent before applying the incoming range. Both halves of the destination image come from the same predecessor the source computed its delta against. allow_partial is then requested only when that chaining actually succeeded on the leader AND on an online secondary -- the transfer lands on the leader and the lvstore mirrors it to the secondary, so an unchained secondary would read zeros wherever the delta did not write. Everything short of that logs the reason and sends a full transfer, which is always correct. The fork also gates the bitmap path on the snapshot's dirty generation being complete and falls back to a full transfer on its own, so the two checks are independent. Nodes chained up front are recorded on the task and skipped by the finish step, because adding the same clone entry twice is not idempotent. convert is untouched. The RPC sends the allow_partial key only when opting in, so every existing caller keeps its current wire form. Retention already keeps the COW parent: _KEEP_REPLICATED_INTERNAL and ReplicationPolicy.MIN_KEEP_REPLICATED are both 2, the schedule path passes always_keep_newest=keep, and _prune_internal_snapshots additionally defers a prune whose successor is not yet chained onto it. Pinned with tests rather than changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twice now a case-7 run has ended with no usable evidence. The fail-back poll calls sbctl on the mgmt node; when that returned rc=1 (a transient CP blip -- `sbctl pool list` worked again minutes later) the exception propagated out of the poll loop, skipped collect_xfer_timing, and left the PREVIOUS run's xfer_timing_case7_failback.log in place. Downloading it produced a file byte-identical to the earlier run -- same five volume ids, same 340.93/1505.63/ 2073.55/2583.53/12.38s durations -- which reads exactly like fresh data until you notice the new phases are missing. Two changes: the poll retries a failed CP call instead of ending the case (the deadline still bounds it), and collection moved into a `finally` so the bundle survives any failure in the loop. The bundle is the point of an instrumented run; losing it costs the whole run.
The iterative snapshots ARE the endgame. They were being taken from the
moment commit ran, so a volume waiting its turn behind 19 others held an
ageing round-1 snapshot, and the "round" that followed measured the wait
rather than the transfer (run 20260828_124859: round 1 growing 340s ->
1505s -> 2073s -> 2584s across successive volumes, while the one volume
that never queued finished in 12.4s).
A cutover now costs the cluster nothing until it can finish quickly:
* commit only enqueues the task -- no snapshot, no round in flight
* the runner waits until ordinary replication has the volume within
REPL_CUTOVER_ENDGAME_LAG_SEC (50s) of the source, suspending without
burning a retry and pushing its deadline out, because catching up is
progress rather than a stall
* only then does it claim the lvstore, and every round runs under that
claim, down to the freeze
So the exclusive window is the tail -- the tight rounds and the freeze --
and the bulk catch-up rides the cadence that was already running.
This removes the "open rounds" of the previous attempt along with
ready_for_exclusive and REPL_CUTOVER_EXCLUSIVE_ENTRY_FACTOR: with the
claim taken before round 1, a round without the lvstore cannot happen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ove duplicate REPL_CUTOVER_ACTIVE_POLL_SEC constant
…ack cutover on volumes with no reverse-direction replication history
…xit inline wait immediately on No process transfer state
…REPL_FAILBACK_RETIRE_ORIGINAL_BEFORE_CUTOVER (off) to stop blob-id reuse breaking cutover delta writes
… is kept, so the create probe cannot silently adopt the original's bdev/blobid
…o CSI device lookup finds the failed-back volume
… operator signals for migration and failback
…ity so shared-subsystem cutovers can claim their preserved nsid
…ginal's uuid and nguid so connected clients keep their multipath heads across cutover
…amespace so restaged clients' multipath heads survive the cutover
…bsystem so a late preconnect stalls IO instead of severing a mounted namespace
… still-alive source cannot keep serving superseded data
…ce retirement makes it redundant
…ion-policy-snapshot in cli-reference.yaml — cli.py was hand-edited and regeneration also restores the missing snapshot dispatch
…he import side effect crashed green CI runs at interpreter shutdown
…he runner requires transfer_state=Done on the return since the 2026-08-22 hardening
geoffrey1330
force-pushed
the
retain_source_lvolID
branch
from
September 4, 2026 11:41
ebc04df to
0a1fc38
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ping new clone UUID in FN_REPLICATION_FINAL