diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index e5b9764e..c4e19d73 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -76,12 +76,40 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr ## Failure Domains and Erasure Coding Schemes -The number of failure domains should match the data protection goal: +The number of failure domains decides two things: whether a cluster can activate with failure domains enabled, +and how much simultaneous node or domain loss it absorbs afterward. -| Goal | Recommendation | -|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------| -| Survive one full domain outage | At least `parity chunks + 1` distinct failure domains | -| Survive one full domain outage plus one further node or drive failure elsewhere | Erasure coding scheme with two parity chunks (e.g., `1+2`, `2+2`) and at least as many domains as data chunks | +!!! important + Fresh activation requires **at least `parity chunks + 2` distinct failure domains**. Two domains are never + enough, at any parity-chunk count. With only `parity chunks + 1` domains the layout has no spare host left, so + the very next node addition or removal strands a failover path with nowhere valid to go. Below the minimum, + activation is refused. Either failure domains have to be disabled, or hosts in further domains have to be + added first. A reactivation of an existing layout is only warned about, never blocked. + +A stripe consists of `data chunks + parity chunks` chunks, and placement spreads them as evenly as the available +domains allow. How much loss an activated cluster absorbs follows from that spread: + +- With **`data chunks + parity chunks` domains or more**, every domain holds at most one chunk of a stripe. The + **complete loss of up to `parity chunks` domains** at once is then tolerated. This is the same guarantee as + running without failure domains, scoped to whole domains instead of individual nodes. +- With **fewer domains than `data chunks + parity chunks`**, at least one domain holds more than one chunk. Each + domain contributes at most `⌈(data chunks + parity chunks) / domains⌉` to a risk budget of `parity chunks`. A + domain that already has that many nodes down has spent its whole contribution, so further nodes in the same + domain cost nothing extra. A node in a different domain spends a fresh share of the same budget. Any combination + is tolerated while the summed contributions stay within `parity chunks`. + +| Domains | 1+1 | 2+1 | 4+1 | 1+2 | 2+2 | 4+2 | +|------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------| +| 2 or fewer | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | +| 3 | 1 whole domain | 1 whole domain | 1 node total | cannot activate | cannot activate | cannot activate | +| 4 | 1 whole domain | 1 whole domain | 1 node total | 2 whole domains | 2 whole domains | 1 whole domain | +| 5 | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 1 whole domain | +| 6+ | 1 whole domain | 1 whole domain | 1 whole domain | 2 whole domains | 2 whole domains | 2 whole domains | + +"1 whole domain" means every node of one domain, whatever its size, can go down at once and the cluster stays +available. The same budget spent as individual nodes, one per domain across that many domains, is tolerated +identically. A cell short of a whole domain (`4+1` on three or four domains) means only that many individual nodes +anywhere in the cluster, not a domain's worth. The high-availability journal requires at least **four** journal copies on failure-domain clusters (instead of three), even with a single parity chunk. With three copies and two domains, one domain would hold two copies and @@ -93,14 +121,62 @@ A host's failure domain cannot be changed while the host is part of the cluster. requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This prevents accidental topology changes that would silently invalidate the placement of existing data. +## When to Remove a Node + +Node removal takes a host out of the cluster permanently. It serves two purposes: retiring a **failed host** that +is not coming back, and **shrinking** a cluster that has more storage nodes than it needs. A node that is offline +or unreachable can still be removed, so removal is the intended way to resolve a dead host rather than a last +resort. + +Leaving a dead node in the cluster is the case to avoid. Domain balance counts every host that has not been +removed, whatever its state, so a dead node still occupies its domain's slot while serving nothing. The cluster +stays degraded, and the balance rules then close the door behind it: + +- The first dead host in a domain can still be removed. Its domain drops to one host below the others, which the + ±1 rule permits. +- If a **second** host in that same domain fails before the first is removed, only one of the two can go. Removing + the second would leave that domain two hosts below the others and, on a three-host domain, below the two-host + floor. Admission refuses it. + +At that point the degradation cannot be cleared by removal at all, and hosts have to be added before the cluster +can be repaired. Remove a failed host promptly, then restore the balance one of two ways: + +- **Add a replacement host** in the same failure domain, returning the split to what it was. +- **Rebalance** by removing hosts from the other domains until every domain is within ±1 again. + +The same reasoning applies to planned shrinking: reduce the domains evenly rather than emptying one, so the +cluster never sits closer to the floor than it has to. + +A removal is admitted only when all of the following hold: + +- The node holds **no volumes and no snapshots**. Migrate or delete them first; removal does not move them. +- The resulting per-domain host split stays within the **±1 rule** and keeps at least **two hosts per domain**. +- Every failover path the node hosts for another volume store has a valid host-disjoint target to move to, after + the splice fallback described in [The Placement Contract](#the-placement-contract). +- The cluster has enough node headroom above `data chunks + parity chunks` to absorb the loss, counting hosts that + are already not online. + +## Journal Copy Replacement on Removal + +Removing a node affects every journal redundancy set that referenced the departed node's journal copy. One +replacement member is picked per set, and every host running a local instance of that set applies the same +decision, so the membership stays identical on all of them. + +The replacement is picked with the same domain-balance goal as the original placement. A candidate from the +**same failure domain** as the departed node is preferred, which leaves the set's domain distribution as it was +instead of reshuffling it. The preference is best-effort, not a requirement: if no same-domain candidate is free, a +cross-domain one is used, and the removal is never blocked over it. Failover-path relocation works in the opposite +direction (see [The Placement Contract](#the-placement-contract)), where a cross-domain target is sometimes +mandatory and the removal is refused without one. + ## Recovery Behavior Failure domains also change how the cluster recovers from large outages: -- An outage confined to one domain (up to and including every node of the domain) keeps the cluster **degraded - but serving**. The cluster is not suspended. -- With two parity chunks, the cluster additionally tolerates the loss of one entire domain **plus** one further - node or device outage in exactly one other domain. +- An outage confined to one domain keeps the cluster **degraded but serving**, as long as that domain's + worst-case contribution fits the parity budget of the sizing table above. The cluster is not suspended. +- With two parity chunks and at least `data chunks + parity chunks` domains, the loss of one entire domain + **plus** one further node or device outage in exactly one other domain is tolerated as well. - When a whole domain returns from an outage (for example, after a rack power loss), its nodes are restarted **in parallel** instead of strictly one-by-one, substantially shortening the recovery of large domains. diff --git a/docs/kubernetes/operations/cluster/failure-domains.md b/docs/kubernetes/operations/cluster/failure-domains.md index c9b62825..9b5bec22 100644 --- a/docs/kubernetes/operations/cluster/failure-domains.md +++ b/docs/kubernetes/operations/cluster/failure-domains.md @@ -169,6 +169,17 @@ In practice, this means workers are added in whole rounds. On a balanced cluster and the next worker has to go to a different one. The full rules are in [Failure Domains: Balance Rules](../../../architecture/concepts/failure-domains.md#balance-rules). +Removal is meant to take a worker out of the cluster, either because it has failed for good or because the cluster +is being shrunk. A failed worker that is left in place still counts toward its domain's host total while serving +nothing, and if a second worker in the same domain fails before the first is removed, only one of the two can be +removed at all. See +[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node). + +Removing a node also triggers journal-copy replacement on every journal redundancy set that referenced its +journal copy. Failover-path relocation can require a cross-domain target. The journal replacement instead prefers +a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it was. See +[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). + For the mechanics of adding the workers themselves, see [Expanding a Storage Cluster](../scaling/expanding-storage-cluster.md), and for taking one out, [Removing a Storage Node](../storage-nodes/removing-a-storage-node.md). diff --git a/docs/non-kubernetes/operations/cluster/failure-domains.md b/docs/non-kubernetes/operations/cluster/failure-domains.md index ddd328b4..0c3f3595 100644 --- a/docs/non-kubernetes/operations/cluster/failure-domains.md +++ b/docs/non-kubernetes/operations/cluster/failure-domains.md @@ -162,10 +162,24 @@ before any change is made. ## Removing a Storage Node +Removal is meant to take a host out of the cluster, either because it has failed for good or because the cluster is +being shrunk. It is not a way to park a broken host: a failed host that is left in place still counts toward its +domain's host total while serving nothing, and once a second host in the same domain fails, only one of the two can +be removed at all. Remove a failed host promptly and then restore the balance. See +[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node). + Node removal applies the same balance rules (±1, minimum two hosts per domain). In addition, the failover paths hosted by the node being removed are relocated to other nodes. If the path being relocated is the only cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the -primary. If no such node exists, the removal is refused. +primary. An idle host in the required domain is looked for first. If none is free, the path is spliced into an +already-formed pairing instead: a host that currently carries another node's failover path is taken over, and that +path is moved onto the primary being repaired. Only when neither option exists is the removal refused. + +Node removal separately triggers journal-copy replacement on every journal redundancy set that referenced the +departed node's journal copy. This works in the opposite direction from failover-path relocation. The replacement +prefers a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it +was instead of reshuffling it. The preference is best-effort, not a requirement, and never blocks the removal. See +[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal). ## Moving a Host Between Domains