Document journal-copy replacement and correct failure-domain sizing rules

docs/architecture/concepts/failure-domains.md

@@ -76,12 +76,40 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr
 
 ## Failure Domains and Erasure Coding Schemes
 
-The number of failure domains should match the data protection goal:
+The number of failure domains decides two things: whether a cluster can activate with failure domains enabled,
+and how much simultaneous node or domain loss it absorbs afterward.
 
-| Goal                                                                            | Recommendation                                                                                                |
-|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
-| Survive one full domain outage                                                  | At least `parity chunks + 1` distinct failure domains                                                         |
-| Survive one full domain outage plus one further node or drive failure elsewhere | Erasure coding scheme with two parity chunks (e.g., `1+2`, `2+2`) and at least as many domains as data chunks |
+!!! important
+    Fresh activation requires **at least `parity chunks + 2` distinct failure domains**. Two domains are never
+    enough, at any parity-chunk count. With only `parity chunks + 1` domains the layout has no spare host left, so
+    the very next node addition or removal strands a failover path with nowhere valid to go. Below the minimum,
+    activation is refused. Either failure domains have to be disabled, or hosts in further domains have to be
+    added first. A reactivation of an existing layout is only warned about, never blocked.
+
+A stripe consists of `data chunks + parity chunks` chunks, and placement spreads them as evenly as the available
+domains allow. How much loss an activated cluster absorbs follows from that spread:
+
+- With **`data chunks + parity chunks` domains or more**, every domain holds at most one chunk of a stripe. The
+  **complete loss of up to `parity chunks` domains** at once is then tolerated. This is the same guarantee as
+  running without failure domains, scoped to whole domains instead of individual nodes.
+- With **fewer domains than `data chunks + parity chunks`**, at least one domain holds more than one chunk. Each
+  domain contributes at most `⌈(data chunks + parity chunks) / domains⌉` to a risk budget of `parity chunks`. A
+  domain that already has that many nodes down has spent its whole contribution, so further nodes in the same
+  domain cost nothing extra. A node in a different domain spends a fresh share of the same budget. Any combination
+  is tolerated while the summed contributions stay within `parity chunks`.
+
+| Domains    | 1+1             | 2+1             | 4+1             | 1+2             | 2+2             | 4+2             |
+|------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
+| 2 or fewer | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate | cannot activate |
+| 3          | 1 whole domain  | 1 whole domain  | 1 node total    | cannot activate | cannot activate | cannot activate |
+| 4          | 1 whole domain  | 1 whole domain  | 1 node total    | 2 whole domains | 2 whole domains | 1 whole domain  |
+| 5          | 1 whole domain  | 1 whole domain  | 1 whole domain  | 2 whole domains | 2 whole domains | 1 whole domain  |
+| 6+         | 1 whole domain  | 1 whole domain  | 1 whole domain  | 2 whole domains | 2 whole domains | 2 whole domains |
+
+"1 whole domain" means every node of one domain, whatever its size, can go down at once and the cluster stays
+available. The same budget spent as individual nodes, one per domain across that many domains, is tolerated
+identically. A cell short of a whole domain (`4+1` on three or four domains) means only that many individual nodes
+anywhere in the cluster, not a domain's worth.
 
 The high-availability journal requires at least **four** journal copies on failure-domain clusters (instead of
 three), even with a single parity chunk. With three copies and two domains, one domain would hold two copies and
@@ -93,14 +121,62 @@ A host's failure domain cannot be changed while the host is part of the cluster.
 requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This
 prevents accidental topology changes that would silently invalidate the placement of existing data.
 
+## When to Remove a Node
+
+Node removal takes a host out of the cluster permanently. It serves two purposes: retiring a **failed host** that
+is not coming back, and **shrinking** a cluster that has more storage nodes than it needs. A node that is offline
+or unreachable can still be removed, so removal is the intended way to resolve a dead host rather than a last
+resort.
+
+Leaving a dead node in the cluster is the case to avoid. Domain balance counts every host that has not been
+removed, whatever its state, so a dead node still occupies its domain's slot while serving nothing. The cluster
+stays degraded, and the balance rules then close the door behind it:
+
+- The first dead host in a domain can still be removed. Its domain drops to one host below the others, which the
+  ±1 rule permits.
+- If a **second** host in that same domain fails before the first is removed, only one of the two can go. Removing
+  the second would leave that domain two hosts below the others and, on a three-host domain, below the two-host
+  floor. Admission refuses it.
+
+At that point the degradation cannot be cleared by removal at all, and hosts have to be added before the cluster
+can be repaired. Remove a failed host promptly, then restore the balance one of two ways:
+
+- **Add a replacement host** in the same failure domain, returning the split to what it was.
+- **Rebalance** by removing hosts from the other domains until every domain is within ±1 again.
+
+The same reasoning applies to planned shrinking: reduce the domains evenly rather than emptying one, so the
+cluster never sits closer to the floor than it has to.
+
+A removal is admitted only when all of the following hold:
+
+- The node holds **no volumes and no snapshots**. Migrate or delete them first; removal does not move them.
+- The resulting per-domain host split stays within the **±1 rule** and keeps at least **two hosts per domain**.
+- Every failover path the node hosts for another volume store has a valid host-disjoint target to move to, after
+  the splice fallback described in [The Placement Contract](#the-placement-contract).
+- The cluster has enough node headroom above `data chunks + parity chunks` to absorb the loss, counting hosts that
+  are already not online.
+
+## Journal Copy Replacement on Removal
+
+Removing a node affects every journal redundancy set that referenced the departed node's journal copy. One
+replacement member is picked per set, and every host running a local instance of that set applies the same
+decision, so the membership stays identical on all of them.
+
+The replacement is picked with the same domain-balance goal as the original placement. A candidate from the
+**same failure domain** as the departed node is preferred, which leaves the set's domain distribution as it was
+instead of reshuffling it. The preference is best-effort, not a requirement: if no same-domain candidate is free, a
+cross-domain one is used, and the removal is never blocked over it. Failover-path relocation works in the opposite
+direction (see [The Placement Contract](#the-placement-contract)), where a cross-domain target is sometimes
+mandatory and the removal is refused without one.
+
 ## Recovery Behavior
 
 Failure domains also change how the cluster recovers from large outages:
 
-- An outage confined to one domain (up to and including every node of the domain) keeps the cluster **degraded
-  but serving**. The cluster is not suspended.
-- With two parity chunks, the cluster additionally tolerates the loss of one entire domain **plus** one further
-  node or device outage in exactly one other domain.
+- An outage confined to one domain keeps the cluster **degraded but serving**, as long as that domain's
+  worst-case contribution fits the parity budget of the sizing table above. The cluster is not suspended.
+- With two parity chunks and at least `data chunks + parity chunks` domains, the loss of one entire domain
+  **plus** one further node or device outage in exactly one other domain is tolerated as well.
 - When a whole domain returns from an outage (for example, after a rack power loss), its nodes are restarted **in
   parallel** instead of strictly one-by-one, substantially shortening the recovery of large domains.
 

docs/kubernetes/operations/cluster/failure-domains.md

@@ -169,6 +169,17 @@ In practice, this means workers are added in whole rounds. On a balanced cluster
 and the next worker has to go to a different one. The full rules are in
 [Failure Domains: Balance Rules](../../../architecture/concepts/failure-domains.md#balance-rules).
 
+Removal is meant to take a worker out of the cluster, either because it has failed for good or because the cluster
+is being shrunk. A failed worker that is left in place still counts toward its domain's host total while serving
+nothing, and if a second worker in the same domain fails before the first is removed, only one of the two can be
+removed at all. See
+[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node).
+
+Removing a node also triggers journal-copy replacement on every journal redundancy set that referenced its
+journal copy. Failover-path relocation can require a cross-domain target. The journal replacement instead prefers
+a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it was. See
+[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal).
+
 For the mechanics of adding the workers themselves, see
 [Expanding a Storage Cluster](../scaling/expanding-storage-cluster.md), and for taking one out,
 [Removing a Storage Node](../storage-nodes/removing-a-storage-node.md).

docs/non-kubernetes/operations/cluster/failure-domains.md

@@ -162,10 +162,24 @@ before any change is made.
 
 ## Removing a Storage Node
 
+Removal is meant to take a host out of the cluster, either because it has failed for good or because the cluster is
+being shrunk. It is not a way to park a broken host: a failed host that is left in place still counts toward its
+domain's host total while serving nothing, and once a second host in the same domain fails, only one of the two can
+be removed at all. Remove a failed host promptly and then restore the balance. See
+[When to Remove a Node](../../../architecture/concepts/failure-domains.md#when-to-remove-a-node).
+
 Node removal applies the same balance rules (±1, minimum two hosts per domain). In addition, the failover paths
 hosted by the node being removed are relocated to other nodes. If the path being relocated is the only
 cross-domain path of its volume store, the replacement node **must** be in a different failure domain than the
-primary. If no such node exists, the removal is refused.
+primary. An idle host in the required domain is looked for first. If none is free, the path is spliced into an
+already-formed pairing instead: a host that currently carries another node's failover path is taken over, and that
+path is moved onto the primary being repaired. Only when neither option exists is the removal refused.
+
+Node removal separately triggers journal-copy replacement on every journal redundancy set that referenced the
+departed node's journal copy. This works in the opposite direction from failover-path relocation. The replacement
+prefers a candidate from the departed node's own failure domain, which leaves the set's domain distribution as it
+was instead of reshuffling it. The preference is best-effort, not a requirement, and never blocks the removal. See
+[Journal Copy Replacement on Removal](../../../architecture/concepts/failure-domains.md#journal-copy-replacement-on-removal).
 
 ## Moving a Host Between Domains