From 1433b795b797bdaf96e65e54d101d9b2fbafd5bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micka=C3=ABl=20Misbach?= Date: Wed, 5 Aug 2026 15:40:24 +0200 Subject: [PATCH 1/2] [docs] Add CRDB survivability configuration --- docs/architecture/index.md | 9 ++-- docs/operations/.nav.yml | 1 + docs/operations/pooling-crdb.md | 8 +++- docs/operations/survivability-crdb.md | 67 +++++++++++++++++++++++++++ 4 files changed, 81 insertions(+), 4 deletions(-) create mode 100644 docs/operations/survivability-crdb.md diff --git a/docs/architecture/index.md b/docs/architecture/index.md index a71567d11..364110910 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -48,9 +48,12 @@ See [Additional requirements](../operations/pooling.md#additional-requirements). One of the primary design considerations of the DSS is to be very resilient to failures. This resiliency is obtained primarily from the behavior of the -underlying CockroachDB database technology and how we configure it. The diagram -below shows the result of failures (bringing a node down for maintenance, or -having an entire USS go down) from different starting points, assuming 3 replicas. +underlying CockroachDB database technology and how we configure it. To meet these +survivability objectives, see [CockroachDB Pool Survivability](../operations/survivability-crdb.md) for how to configure +the CockroachDB pool with replica placement constraints. + +The diagram below shows the result of failures (bringing a node down for maintenance, +or having an entire USS go down) from different starting points, assuming 3 replicas. ![Survivability diagram](../assets/generated/survivability_3x2.png) diff --git a/docs/operations/.nav.yml b/docs/operations/.nav.yml index 5df31ef53..90ede316a 100644 --- a/docs/operations/.nav.yml +++ b/docs/operations/.nav.yml @@ -3,6 +3,7 @@ nav: - "Certificates management (Yugabyte)": certificates-management.md - "Pooling (Yugabyte)": pooling.md - "Pooling (CockroachDB)": pooling-crdb.md + - "Pool Survivability (CockroachDB)": survivability-crdb.md - "Monitoring": monitoring.md - "Health checks": healthchecks.md - "Migrations": migrations.md diff --git a/docs/operations/pooling-crdb.md b/docs/operations/pooling-crdb.md index 4e24e5fab..a3ba33715 100644 --- a/docs/operations/pooling-crdb.md +++ b/docs/operations/pooling-crdb.md @@ -124,7 +124,8 @@ following those instructions. `--certs-dir` and `--ca-key` flags. - Do not specify `--insecure` - The ordering of the `--locality` flag keys must be the same across all - CockroachDB nodes in the cluster. + CockroachDB nodes in the cluster. See [CockroachDB Pool Survivability](survivability-crdb.md) + for details on setting up these values and configuring replication zone constraints. - All DSS instances in the same cluster must point their ntpd at the same NTP Servers. [CockroachDB recommends](https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#considerations) @@ -210,6 +211,11 @@ to scale down the Statefulset and that removes the last node first (ex: 1. If there are unhealthy nodes please investigate and fix them so that the ranges can return to a healthy state +1. Identify whether the replication zone constraints should be adapted + accordingly. This might be the case e.g. when all nodes of an USS are leaving + the pool. See [CockroachDB Pool Survivability](survivability-crdb.md) for + details on setting up these values. + 1. Identify the node id we intend to decommission from the previous commands then decommission them. The following command assumes that `cockroachdb-0` is not targeted for decommission otherwise select a different instance to diff --git a/docs/operations/survivability-crdb.md b/docs/operations/survivability-crdb.md new file mode 100644 index 000000000..9e1368db4 --- /dev/null +++ b/docs/operations/survivability-crdb.md @@ -0,0 +1,67 @@ +# CockroachDB Pool Survivability + +This document describes how to meet the survivability and high-availability objectives of a DSS pool running on CockroachDB (as described in the [Survivability section of the Architecture Overview](../architecture/index.md#Survivability)). + +For example, in a standard 3-USS pool running 2 CockroachDB nodes per USS (6 nodes total): +* If we do not restrict where data replicas are placed, CockroachDB might place two of the three replicas of a range on nodes belonging to a single USS. +* If that USS suffers an outage or goes down for maintenance, we would lose two replicas at once. This breaks the Raft quorum (since only 1 of 3 replicas remains active), causing that range to become unavailable. +* To prevent this and meet survivability objectives, we must configure the CockroachDB pool to place at least one replica per USS. This guarantees that even if a full USS goes down, 2 out of 3 replicas remain active on the other USSs, maintaining quorum and ensuring uninterrupted DSS operations. + +To achieve this, we: +1. Start each CockroachDB node with a specific locality embedding its USS and node identifier. +2. Configure replication zones using the CockroachDB `ALTER RANGE` SQL command to enforce placement constraints across the pool. + +--- + +## 1. Setting Node Locality + +When starting each CockroachDB node, you must configure its `--locality` flag to specify which USS and node it represents. The locality must use first at least the `uss` key, and optionally second the `node` key. + +!!! danger "Ordering Constraint" + The ordering of the `--locality` flag keys must be exactly the same across all CockroachDB nodes in the cluster (e.g., `uss` first, then `node`). Mixing the order (e.g., `node` then `uss` on some nodes) will cause CockroachDB to treat them as incompatible locality hierarchies and fail to apply constraints correctly. + +### Flag Format + +```shell +--locality=uss=,node= +# or +--locality=uss= +``` + +Where: +* `` is a unique identifier for the USS organization (e.g., `uss1`, `uss2`, `uss3`). +* `` is a unique identifier for the node within that USS (e.g., `node-0`, `node-1`). + +### Example Configuration for a 3-USS Pool + +* **USS 1 (uss1)**: + * Node 0: `--locality=uss=uss1,node=node-0` + * Node 1: `--locality=uss=uss1,node=node-1` +* **USS 2 (uss2)**: + * Node 0: `--locality=uss=uss2,node=node-0` + * Node 1: `--locality=uss=uss2,node=node-1` +* **USS 3 (uss3)**: + * Node 0: `--locality=uss=uss3,node=node-0` + * Node 1: `--locality=uss=uss3,node=node-1` + + +## 2. Configuring Replication Constraints (`ALTER RANGE`) + +By default, CockroachDB automatically distributes replicas to optimize resource usage and load. To enforce the "one replica per USS" survivability rule, you must manually define replication zone constraints using the `ALTER RANGE default CONFIGURE ZONE` SQL command. + +The `default` range is the cluster-wide catch-all. Any database or table created within the DSS (including the RID and SCD tables) that does not have its own specific zone configuration will inherit these default placement constraints. + +### The `ALTER RANGE` SQL Statement + +To configure a 3-replica cluster where exactly one replica lives on each of the three USSs: + +```sql +ALTER RANGE default CONFIGURE ZONE USING + num_replicas = 3, + constraints = '{"+uss=uss1": 1, "+uss=uss2": 1, "+uss=uss3": 1}'; +``` + +### Explanation of the Parameters: +* `num_replicas = 3`: Tells CockroachDB to keep 3 copies of each range. +* `constraints`: A JSON object specifying per-replica constraints. Do note that specifying this might not always be necessary depending on your deployment. + * `"+uss=uss1": 1` tells CockroachDB that exactly one replica must be located on nodes matching the locality `uss=uss1`. From 26e27f1d27465a9a957d95fa61bdc53a3546b54b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micka=C3=ABl=20Misbach?= Date: Wed, 26 Aug 2026 10:26:15 +0200 Subject: [PATCH 2/2] Clarify doc scope: only cover default case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Mickaël Misbach --- docs/operations/survivability-crdb.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/operations/survivability-crdb.md b/docs/operations/survivability-crdb.md index 9e1368db4..4e1eac6a6 100644 --- a/docs/operations/survivability-crdb.md +++ b/docs/operations/survivability-crdb.md @@ -15,7 +15,22 @@ To achieve this, we: ## 1. Setting Node Locality -When starting each CockroachDB node, you must configure its `--locality` flag to specify which USS and node it represents. The locality must use first at least the `uss` key, and optionally second the `node` key. +When starting each CockroachDB node, you must configure its `--locality` flag to specify which USS and node it represents. For the purpose of ensuring survivability of the cluster, it is important that all USSes align of the structure of their locality settings. +We recommend that the locality use first at least the `uss` key, and optionally second the `node` key. +Because CockroachDB will distribute range replicas using values of locality keys, the order of the keys defining the hierarchy to do so, if there is some existing locality set preventing this to be achieved, it is important that: +- all USSes define a specific key whose value is specific to them (e.g. `uss=uss1`); +- and that the number of total replicas set is covers at least the total number of combination of keys of to the USS key. + +Example: with three USSs each with two nodes define the following localities: +- `region=east,uss=uss1,node=uss1_node1` +- `region=west,uss=uss1,node=uss1_node2` +- `region=east,uss=uss2,node=uss2_node1` +- `region=west,uss=uss2,node=uss2_node2` +- `region=east,uss=uss3,node=uss3_node3` +- `region=west,uss=uss3,node=uss3_node3` +Then, the total number of replicas should be set to at least 6 (2 `region` * 3 `uss`). + +If, for some reason specific to your deployment, this is not possible to achieve, you will need to configure specific range constraints on your deployment to ensure survivability. This is however not within the scope of this documentation. !!! danger "Ordering Constraint" The ordering of the `--locality` flag keys must be exactly the same across all CockroachDB nodes in the cluster (e.g., `uss` first, then `node`). Mixing the order (e.g., `node` then `uss` on some nodes) will cause CockroachDB to treat them as incompatible locality hierarchies and fail to apply constraints correctly. @@ -53,15 +68,15 @@ The `default` range is the cluster-wide catch-all. Any database or table created ### The `ALTER RANGE` SQL Statement -To configure a 3-replica cluster where exactly one replica lives on each of the three USSs: +To configure a 3-replica cluster where exactly one replica lives on each of the three USSs, assuming locality is set on each node as described above: ```sql ALTER RANGE default CONFIGURE ZONE USING num_replicas = 3, - constraints = '{"+uss=uss1": 1, "+uss=uss2": 1, "+uss=uss3": 1}'; + num_replicas = 3; ``` ### Explanation of the Parameters: * `num_replicas = 3`: Tells CockroachDB to keep 3 copies of each range. -* `constraints`: A JSON object specifying per-replica constraints. Do note that specifying this might not always be necessary depending on your deployment. - * `"+uss=uss1": 1` tells CockroachDB that exactly one replica must be located on nodes matching the locality `uss=uss1`. + +Do note that outside of the default case described in this documentation, you will need to adjust the above SQL query by configuring different ranges and/or setting specific constraints. This is however not within the scope of this documentation.