From 89067beccdace4b88731d15a138b63de38c7fb8b Mon Sep 17 00:00:00 2001 From: wmousa Date: Wed, 26 Aug 2026 08:09:27 +0200 Subject: [PATCH 1/4] Document journal-copy replacement's failure-domain preference on node removal Failure-domain node removal already documented that failover-path relocation can require a cross-domain target and refuses the removal if none exists. It never documented the journal (JM) redundancy set's own replacement behavior, which runs at the same time but in the opposite direction: the control plane prefers a replacement from the departed node's own failure domain, to leave the set's domain distribution unchanged rather than reshuffled -- a best-effort preference, never a removal blocker. Adds a new "Journal Copy Replacement on Removal" section to the architecture concept page, and cross-references it from both the Kubernetes and non-Kubernetes "failure domain" operations pages, next to their existing failover-path relocation coverage. Verified with scripts/quality-gate.sh (spelling, terminology, american, prose, voice, punctuation, syntax) -- clean for all three files. --- docs/architecture/concepts/failure-domains.md | 13 +++++++++++++ .../operations/cluster/failure-domains.md | 5 +++++ .../operations/cluster/failure-domains.md | 7 +++++++ 3 files changed, 25 insertions(+) diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index e5b9764e..9fa227b9 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -93,6 +93,19 @@ A host's failure domain cannot be changed while the host is part of the cluster. requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This prevents accidental topology changes that would silently invalidate the placement of existing data. +## Journal Copy Replacement on Removal + +Removing a node also affects every journal redundancy set that included the departed node's journal copy. Each +surviving node running a local copy of such a set picks a replacement member before the departed journal is +retired. + +The replacement is chosen with the same domain-balance goal as the original placement: a candidate from the +**same failure domain** as the departed node is preferred, so the set's domain distribution is left unchanged +rather than reshuffled. This is a best-effort preference, not a hard requirement: if no same-domain candidate is +available, a cross-domain one is used instead, and the removal itself is never blocked by it. This is the opposite +direction from failover-path relocation (see [The Placement Contract](#the-placement-contract)), which sometimes +requires a cross-domain target and refuses the removal if none exists. + ## Recovery Behavior Failure domains also change how the cluster recovers from large outages: diff --git a/docs/kubernetes/operations/cluster/failure-domains.md b/docs/kubernetes/operations/cluster/failure-domains.md index c9b62825..946a5e4a 100644 --- a/docs/kubernetes/operations/cluster/failure-domains.md +++ b/docs/kubernetes/operations/cluster/failure-domains.md @@ -169,6 +169,11 @@ In practice, this means workers are added in whole rounds. On a balanced cluster and the next worker has to go to a different one. The full rules are in [Failure Domains: Balance Rules](../../../architecture/concepts/failure-domains.md#balance-rules). +Removing a node also triggers journal-copy replacement on every journal redundancy set that included its journal +copy. Unlike failover-path relocation, which can require a cross-domain target, the journal replacement *prefers* +a candidate from the departed node's own failure domain, to leave the set's domain distribution unchanged. See +[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). + For the mechanics of adding the workers themselves, see [Expanding a Storage Cluster](../scaling/expanding-storage-cluster.md), and for taking one out, [Removing a Storage Node](../storage-nodes/removing-a-storage-node.md). diff --git a/docs/non-kubernetes/operations/cluster/failure-domains.md b/docs/non-kubernetes/operations/cluster/failure-domains.md index ddd328b4..c6a93121 100644 --- a/docs/non-kubernetes/operations/cluster/failure-domains.md +++ b/docs/non-kubernetes/operations/cluster/failure-domains.md @@ -167,6 +167,13 @@ hosted by the node being removed are relocated to other nodes. If the path being cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the primary. If no such node exists, the removal is refused. +Node removal separately triggers journal-copy replacement on every journal redundancy set that included the +departed node's journal copy. This works in the opposite direction from failover-path relocation: the replacement +prefers a candidate from the departed node's own failure domain, so the set's domain distribution is left +unchanged rather than reshuffled. It is a best-effort preference, not a hard requirement, and never blocks the +removal. See +[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). + ## Moving a Host Between Domains A host's failure domain is immutable. Re-adding a host or one of its node slots with a different domain label is From ddf2989c0d94e1becfbc4d4f8b9e01b3d81f5c9c Mon Sep 17 00:00:00 2001 From: wmousa Date: Wed, 26 Aug 2026 17:44:38 +0200 Subject: [PATCH 2/4] Correct FD activation minimum and node-removal refusal, add domain sizing table Two claims in the failure-domain docs no longer matched the code: - "At least parity chunks + 1 domains" for surviving a domain outage was the bare rotation-correctness floor, not the actual activation gate. Fresh activation now hard-requires parity chunks + 2 distinct domains (2 domains is never sufficient, at any parity-chunk count) so a later single add/remove always has a spare host to relocate onto, rather than stranding a failover path immediately. - "If no such node exists, the removal is refused" skipped a fallback: the control plane now splices a stranded failover path into an already-formed pairing (rebuilding that host's own path elsewhere) before actually refusing the removal. Replaces the old goal/recommendation table with a domain-count x erasure-coding-scheme sizing table showing exactly how much simultaneous node or domain loss each combination tolerates, derived from the same per-domain risk-budget formula the node-shutdown capacity check uses (chunks_per_domain = ceil((ndcs+npcs) / domains), capped contribution per domain, summed against a parity-chunks budget). Verified with scripts/quality-gate.sh (spelling, terminology, american, prose, voice, punctuation, syntax) -- clean for both files; the syntax gate's remaining failures are pre-existing broken CLI-reference links in unrelated, untouched files. --- docs/architecture/concepts/failure-domains.md | 37 ++++++++++++++++--- .../operations/cluster/failure-domains.md | 4 +- 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index 9fa227b9..cf2d9790 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -76,12 +76,39 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr ## Failure Domains and Erasure Coding Schemes -The number of failure domains should match the data protection goal: +The number of failure domains determines both whether a cluster can activate with failure domains enabled at all, +and how much simultaneous node or domain loss it can then absorb. -| Goal | Recommendation | -|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------| -| Survive one full domain outage | At least `parity chunks + 1` distinct failure domains | -| Survive one full domain outage plus one further node or drive failure elsewhere | Erasure coding scheme with two parity chunks (e.g., `1+2`, `2+2`) and at least as many domains as data chunks | +!!! important + Fresh activation requires **at least `parity chunks + 2` distinct failure domains**. Two domains are never + enough, at any parity-chunk count: with only `parity chunks + 1` domains the layout has no spare host, so the + very next node add or remove would strand a failover path with nowhere valid to go. Below the minimum, + activation is refused. Disable failure domains, or add hosts in further domains first. + +Once activated, tolerance for simultaneous node or domain outages follows from how evenly a stripe's +`data chunks + parity chunks` spread across the available domains: + +- With **`data chunks + parity chunks` domains or more**, each domain holds at most one chunk per stripe, so the + cluster tolerates the **complete loss of up to `parity chunks` domains** at once (the same guarantee as running + without failure domains, now scoped to whole domains instead of individual nodes). +- With **fewer domains than `data chunks + parity chunks`**, at least one domain necessarily holds more than one + chunk. A domain already carrying `⌈(data chunks + parity chunks) / domains⌉` down nodes has spent its entire + worst-case contribution: further nodes going down in that same domain cost nothing extra, but a different + domain going down spends a fresh share of the same budget. Any combination is tolerated as long as the summed + worst-case contribution of the affected domains stays within `parity chunks`. + +| Domains | 1+1 | 2+1 | 4+1 | 1+2 | 2+2 | 4+2 | +|---------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------| +| 1 – 2 | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | +| 3 | 1 whole domain | 1 whole domain | 1 node total | cannot activate | cannot activate | cannot activate | +| 4 | 1 whole domain | 1 whole domain | 1 node total | 2 whole domains | 2 whole domains | 1 whole domain | +| 5 | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 1 whole domain | +| 6+ | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 2 whole domains | + +"1 whole domain" means every node in one domain, regardless of the domain's size, can go down at once and the +cluster stays available. Spending the identical budget as individual nodes spread one-per-domain across that many +domains is tolerated the same way. A cell short of a whole domain (`4+1` at 3 – 4 domains) instead means only that +many individual nodes total, anywhere in the cluster, survive at once, not a domain's worth. The high-availability journal requires at least **four** journal copies on failure-domain clusters (instead of three), even with a single parity chunk. With three copies and two domains, one domain would hold two copies and diff --git a/docs/non-kubernetes/operations/cluster/failure-domains.md b/docs/non-kubernetes/operations/cluster/failure-domains.md index c6a93121..49928c6a 100644 --- a/docs/non-kubernetes/operations/cluster/failure-domains.md +++ b/docs/non-kubernetes/operations/cluster/failure-domains.md @@ -165,7 +165,9 @@ before any change is made. Node removal applies the same balance rules (±1, minimum two hosts per domain). In addition, the failover paths hosted by the node being removed are relocated to other nodes. If the path being relocated is the only cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the -primary. If no such node exists, the removal is refused. +primary. The control plane first looks for an idle host in the right domain. If none is free, it splices the path +into an already-formed pairing instead, rebuilding that host's own path elsewhere to make room, rather than giving +up immediately. Only when neither option exists is the removal refused. Node removal separately triggers journal-copy replacement on every journal redundancy set that included the departed node's journal copy. This works in the opposite direction from failover-path relocation: the replacement From 7f2a6df1b337c62fe338dacb91ef7150ee880334 Mon Sep 17 00:00:00 2001 From: "Christoph Engelbert (noctarius)" Date: Mon, 31 Aug 2026 22:15:04 +0200 Subject: [PATCH 3/4] Style alignment --- docs/architecture/concepts/failure-domains.md | 87 ++++++++++--------- .../operations/cluster/failure-domains.md | 6 +- .../operations/cluster/failure-domains.md | 17 ++-- 3 files changed, 55 insertions(+), 55 deletions(-) diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index cf2d9790..d9f08297 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -76,39 +76,40 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr ## Failure Domains and Erasure Coding Schemes -The number of failure domains determines both whether a cluster can activate with failure domains enabled at all, -and how much simultaneous node or domain loss it can then absorb. +The number of failure domains decides two things: whether a cluster can activate with failure domains enabled, +and how much simultaneous node or domain loss it absorbs afterward. !!! important Fresh activation requires **at least `parity chunks + 2` distinct failure domains**. Two domains are never - enough, at any parity-chunk count: with only `parity chunks + 1` domains the layout has no spare host, so the - very next node add or remove would strand a failover path with nowhere valid to go. Below the minimum, - activation is refused. Disable failure domains, or add hosts in further domains first. - -Once activated, tolerance for simultaneous node or domain outages follows from how evenly a stripe's -`data chunks + parity chunks` spread across the available domains: - -- With **`data chunks + parity chunks` domains or more**, each domain holds at most one chunk per stripe, so the - cluster tolerates the **complete loss of up to `parity chunks` domains** at once (the same guarantee as running - without failure domains, now scoped to whole domains instead of individual nodes). -- With **fewer domains than `data chunks + parity chunks`**, at least one domain necessarily holds more than one - chunk. A domain already carrying `⌈(data chunks + parity chunks) / domains⌉` down nodes has spent its entire - worst-case contribution: further nodes going down in that same domain cost nothing extra, but a different - domain going down spends a fresh share of the same budget. Any combination is tolerated as long as the summed - worst-case contribution of the affected domains stays within `parity chunks`. - -| Domains | 1+1 | 2+1 | 4+1 | 1+2 | 2+2 | 4+2 | -|---------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------| -| 1 – 2 | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | -| 3 | 1 whole domain | 1 whole domain | 1 node total | cannot activate | cannot activate | cannot activate | -| 4 | 1 whole domain | 1 whole domain | 1 node total | 2 whole domains | 2 whole domains | 1 whole domain | -| 5 | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 1 whole domain | -| 6+ | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 2 whole domains | - -"1 whole domain" means every node in one domain, regardless of the domain's size, can go down at once and the -cluster stays available. Spending the identical budget as individual nodes spread one-per-domain across that many -domains is tolerated the same way. A cell short of a whole domain (`4+1` at 3 – 4 domains) instead means only that -many individual nodes total, anywhere in the cluster, survive at once, not a domain's worth. + enough, at any parity-chunk count. With only `parity chunks + 1` domains the layout has no spare host left, so + the very next node addition or removal strands a failover path with nowhere valid to go. Below the minimum, + activation is refused. Either failure domains have to be disabled, or hosts in further domains have to be + added first. A reactivation of an existing layout is only warned about, never blocked. + +A stripe consists of `data chunks + parity chunks` chunks, and placement spreads them as evenly as the available +domains allow. How much loss an activated cluster absorbs follows from that spread: + +- With **`data chunks + parity chunks` domains or more**, every domain holds at most one chunk of a stripe. The + **complete loss of up to `parity chunks` domains** at once is then tolerated. This is the same guarantee as + running without failure domains, scoped to whole domains instead of individual nodes. +- With **fewer domains than `data chunks + parity chunks`**, at least one domain holds more than one chunk. Each + domain contributes at most `⌈(data chunks + parity chunks) / domains⌉` to a risk budget of `parity chunks`. A + domain that already has that many nodes down has spent its whole contribution, so further nodes in the same + domain cost nothing extra. A node in a different domain spends a fresh share of the same budget. Any combination + is tolerated while the summed contributions stay within `parity chunks`. + +| Domains | 1+1 | 2+1 | 4+1 | 1+2 | 2+2 | 4+2 | +|------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------| +| 2 or fewer | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | +| 3 | 1 whole domain | 1 whole domain | 1 node total | cannot activate | cannot activate | cannot activate | +| 4 | 1 whole domain | 1 whole domain | 1 node total | 2 whole domains | 2 whole domains | 1 whole domain | +| 5 | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 1 whole domain | +| 6+ | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 2 whole domains | + +"1 whole domain" means every node of one domain, whatever its size, can go down at once and the cluster stays +available. The same budget spent as individual nodes, one per domain across that many domains, is tolerated +identically. A cell short of a whole domain (`4+1` on three or four domains) means only that many individual nodes +anywhere in the cluster, not a domain's worth. The high-availability journal requires at least **four** journal copies on failure-domain clusters (instead of three), even with a single parity chunk. With three copies and two domains, one domain would hold two copies and @@ -122,25 +123,25 @@ prevents accidental topology changes that would silently invalidate the placemen ## Journal Copy Replacement on Removal -Removing a node also affects every journal redundancy set that included the departed node's journal copy. Each -surviving node running a local copy of such a set picks a replacement member before the departed journal is -retired. +Removing a node affects every journal redundancy set that referenced the departed node's journal copy. One +replacement member is picked per set, and every host running a local instance of that set applies the same +decision, so the membership stays identical on all of them. -The replacement is chosen with the same domain-balance goal as the original placement: a candidate from the -**same failure domain** as the departed node is preferred, so the set's domain distribution is left unchanged -rather than reshuffled. This is a best-effort preference, not a hard requirement: if no same-domain candidate is -available, a cross-domain one is used instead, and the removal itself is never blocked by it. This is the opposite -direction from failover-path relocation (see [The Placement Contract](#the-placement-contract)), which sometimes -requires a cross-domain target and refuses the removal if none exists. +The replacement is picked with the same domain-balance goal as the original placement. A candidate from the +**same failure domain** as the departed node is preferred, which leaves the set's domain distribution as it was +instead of reshuffling it. The preference is best-effort, not a requirement: if no same-domain candidate is free, a +cross-domain one is used, and the removal is never blocked over it. Failover-path relocation works in the opposite +direction (see [The Placement Contract](#the-placement-contract)), where a cross-domain target is sometimes +mandatory and the removal is refused without one. ## Recovery Behavior Failure domains also change how the cluster recovers from large outages: -- An outage confined to one domain (up to and including every node of the domain) keeps the cluster **degraded - but serving**. The cluster is not suspended. -- With two parity chunks, the cluster additionally tolerates the loss of one entire domain **plus** one further - node or device outage in exactly one other domain. +- An outage confined to one domain keeps the cluster **degraded but serving**, as long as that domain's + worst-case contribution fits the parity budget of the sizing table above. The cluster is not suspended. +- With two parity chunks and at least `data chunks + parity chunks` domains, the loss of one entire domain + **plus** one further node or device outage in exactly one other domain is tolerated as well. - When a whole domain returns from an outage (for example, after a rack power loss), its nodes are restarted **in parallel** instead of strictly one-by-one, substantially shortening the recovery of large domains. diff --git a/docs/kubernetes/operations/cluster/failure-domains.md b/docs/kubernetes/operations/cluster/failure-domains.md index 946a5e4a..548a35a5 100644 --- a/docs/kubernetes/operations/cluster/failure-domains.md +++ b/docs/kubernetes/operations/cluster/failure-domains.md @@ -169,9 +169,9 @@ In practice, this means workers are added in whole rounds. On a balanced cluster and the next worker has to go to a different one. The full rules are in [Failure Domains: Balance Rules](../../../architecture/concepts/failure-domains.md#balance-rules). -Removing a node also triggers journal-copy replacement on every journal redundancy set that included its journal -copy. Unlike failover-path relocation, which can require a cross-domain target, the journal replacement *prefers* -a candidate from the departed node's own failure domain, to leave the set's domain distribution unchanged. See +Removing a node also triggers journal-copy replacement on every journal redundancy set that referenced its +journal copy. Failover-path relocation can require a cross-domain target. The journal replacement instead prefers +a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it was. See [Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). For the mechanics of adding the workers themselves, see diff --git a/docs/non-kubernetes/operations/cluster/failure-domains.md b/docs/non-kubernetes/operations/cluster/failure-domains.md index 49928c6a..021464f8 100644 --- a/docs/non-kubernetes/operations/cluster/failure-domains.md +++ b/docs/non-kubernetes/operations/cluster/failure-domains.md @@ -165,15 +165,14 @@ before any change is made. Node removal applies the same balance rules (±1, minimum two hosts per domain). In addition, the failover paths hosted by the node being removed are relocated to other nodes. If the path being relocated is the only cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the -primary. The control plane first looks for an idle host in the right domain. If none is free, it splices the path -into an already-formed pairing instead, rebuilding that host's own path elsewhere to make room, rather than giving -up immediately. Only when neither option exists is the removal refused. - -Node removal separately triggers journal-copy replacement on every journal redundancy set that included the -departed node's journal copy. This works in the opposite direction from failover-path relocation: the replacement -prefers a candidate from the departed node's own failure domain, so the set's domain distribution is left -unchanged rather than reshuffled. It is a best-effort preference, not a hard requirement, and never blocks the -removal. See +primary. An idle host in the required domain is looked for first. If none is free, the path is spliced into an +already-formed pairing instead: a host that currently carries another node's failover path is taken over, and that +path is moved onto the primary being repaired. Only when neither option exists is the removal refused. + +Node removal separately triggers journal-copy replacement on every journal redundancy set that referenced the +departed node's journal copy. This works in the opposite direction from failover-path relocation. The replacement +prefers a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it +was instead of reshuffling it. The preference is best-effort, not a requirement, and never blocks the removal. See [Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). ## Moving a Host Between Domains From 476961a10c21c0d601062a2e67f81dcbc548a2c3 Mon Sep 17 00:00:00 2001 From: wmousa Date: Thu, 3 Sep 2026 10:11:54 +0200 Subject: [PATCH 4/4] Document what node removal is for and the cost of deferring it The failure-domain pages covered the balance rules a removal has to satisfy but never said what removal is for, and the omission hides a trap that only shows up on the second failure. Domain balance counts every host that has not been removed, whatever its state (failure_domain_host_map skips only STATUS_REMOVED and dedicated secondaries), so a dead host left in the cluster still occupies its domain's slot while serving nothing. The first such host can still be removed -- its domain drops one below the others, which the +/-1 rule permits. If a second host in the same domain fails first, only one of the two can go: removing the second would leave that domain two hosts below the others and, on a three-host domain, below the two-host floor, and admission refuses it. The degradation then cannot be cleared by removal at all until hosts are added. Adds a "When to Remove a Node" section to the architecture concept page stating the two intended uses (retiring a failed host, shrinking a cluster), that an offline or unreachable node can still be removed so removal is the intended path for a dead host rather than a last resort, the two-failure trap above, the two ways to restore balance afterwards, and the admission conditions an operator can act on -- no volumes or snapshots on the node, the +/-1 and two-host rules, a valid host-disjoint target for every failover path the node hosts, and node headroom above data+parity counting hosts already not online. Cross-references it from the Kubernetes and non-Kubernetes "failure domain" operations pages, next to their existing removal coverage. Verified with scripts/quality-gate.sh: spelling, terminology, american, prose, voice and punctuation clean. The syntax gate's failures are the same pre-existing broken CLI-reference links in unrelated, untouched files; none of the three files changed here appears among them. --- docs/architecture/concepts/failure-domains.md | 35 +++++++++++++++++++ .../operations/cluster/failure-domains.md | 6 ++++ .../operations/cluster/failure-domains.md | 6 ++++ 3 files changed, 47 insertions(+) diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index d9f08297..c4e19d73 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -121,6 +121,41 @@ A host's failure domain cannot be changed while the host is part of the cluster. requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This prevents accidental topology changes that would silently invalidate the placement of existing data. +## When to Remove a Node + +Node removal takes a host out of the cluster permanently. It serves two purposes: retiring a **failed host** that +is not coming back, and **shrinking** a cluster that has more storage nodes than it needs. A node that is offline +or unreachable can still be removed, so removal is the intended way to resolve a dead host rather than a last +resort. + +Leaving a dead node in the cluster is the case to avoid. Domain balance counts every host that has not been +removed, whatever its state, so a dead node still occupies its domain's slot while serving nothing. The cluster +stays degraded, and the balance rules then close the door behind it: + +- The first dead host in a domain can still be removed. Its domain drops to one host below the others, which the + ±1 rule permits. +- If a **second** host in that same domain fails before the first is removed, only one of the two can go. Removing + the second would leave that domain two hosts below the others and, on a three-host domain, below the two-host + floor. Admission refuses it. + +At that point the degradation cannot be cleared by removal at all, and hosts have to be added before the cluster +can be repaired. Remove a failed host promptly, then restore the balance one of two ways: + +- **Add a replacement host** in the same failure domain, returning the split to what it was. +- **Rebalance** by removing hosts from the other domains until every domain is within ±1 again. + +The same reasoning applies to planned shrinking: reduce the domains evenly rather than emptying one, so the +cluster never sits closer to the floor than it has to. + +A removal is admitted only when all of the following hold: + +- The node holds **no volumes and no snapshots**. Migrate or delete them first; removal does not move them. +- The resulting per-domain host split stays within the **±1 rule** and keeps at least **two hosts per domain**. +- Every failover path the node hosts for another volume store has a valid host-disjoint target to move to, after + the splice fallback described in [The Placement Contract](#the-placement-contract). +- The cluster has enough node headroom above `data chunks + parity chunks` to absorb the loss, counting hosts that + are already not online. + ## Journal Copy Replacement on Removal Removing a node affects every journal redundancy set that referenced the departed node's journal copy. One diff --git a/docs/kubernetes/operations/cluster/failure-domains.md b/docs/kubernetes/operations/cluster/failure-domains.md index 548a35a5..9b5bec22 100644 --- a/docs/kubernetes/operations/cluster/failure-domains.md +++ b/docs/kubernetes/operations/cluster/failure-domains.md @@ -169,6 +169,12 @@ In practice, this means workers are added in whole rounds. On a balanced cluster and the next worker has to go to a different one. The full rules are in [Failure Domains: Balance Rules](../../../architecture/concepts/failure-domains.md#balance-rules). +Removal is meant to take a worker out of the cluster, either because it has failed for good or because the cluster +is being shrunk. A failed worker that is left in place still counts toward its domain's host total while serving +nothing, and if a second worker in the same domain fails before the first is removed, only one of the two can be +removed at all. See +[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node). + Removing a node also triggers journal-copy replacement on every journal redundancy set that referenced its journal copy. Failover-path relocation can require a cross-domain target. The journal replacement instead prefers a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it was. See diff --git a/docs/non-kubernetes/operations/cluster/failure-domains.md b/docs/non-kubernetes/operations/cluster/failure-domains.md index 021464f8..0c3f3595 100644 --- a/docs/non-kubernetes/operations/cluster/failure-domains.md +++ b/docs/non-kubernetes/operations/cluster/failure-domains.md @@ -162,6 +162,12 @@ before any change is made. ## Removing a Storage Node +Removal is meant to take a host out of the cluster, either because it has failed for good or because the cluster is +being shrunk. It is not a way to park a broken host: a failed host that is left in place still counts toward its +domain's host total while serving nothing, and once a second host in the same domain fails, only one of the two can +be removed at all. Remove a failed host promptly and then restore the balance. See +[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node). + Node removal applies the same balance rules (±1, minimum two hosts per domain). In addition, the failover paths hosted by the node being removed are relocated to other nodes. If the path being relocated is the only cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the