feat(cluster): make etcd's heartbeat and election timeouts configurable

roles/github_runner_cluster/README.md

@@ -143,6 +143,22 @@ The limitation: the control server lives on one node. While the bootstrap server
 
 Headscale's database lives on the bootstrap server's k3s data volume, and a static pod cannot use a replicated persistent volume instead, since kubelet runs it with no API server to bind a claim. So the provider requires Litestream: set `github_runner_cluster_headscale_litestream_replica` (see [Backups and a warm standby](#backups-and-a-warm-standby) for the settings), and the pod gets a `litestream-restore` init container and a `litestream` sidecar. A rebuilt bootstrap server then restores the database before Headscale starts, with the same data loss as `headscale_hosted`. The Noise key copy in `keys/` sits on the host, outside the k3s volume. It survives losing that volume but not the host, so supply `github_runner_cluster_headscale_noise_private_key` too if the host itself may be replaced. To run without Litestream, set `github_runner_cluster_headscale_accept_node_local_state: true`, and accept that losing that volume loses every registration: every node must then join again with a new key. There is no standby for this provider; recovery is rebuilding the bootstrap server.
 
+## etcd timing
+
+k3s's embedded etcd runs with a 100 ms heartbeat interval and a 1000 ms election timeout. Where the servers' peer traffic goes through a relay, as it does when each node's container sits behind NAT, round trips and jitter can exceed that: followers miss heartbeats, elections repeat, reads against etcd fail as Unavailable and k3s exits and restarts, which kills the pods on that node. `github_runner_cluster_etcd_heartbeat_interval` and `github_runner_cluster_etcd_election_timeout` (milliseconds, both empty by default so k3s's own timing applies and nothing changes for an existing fleet) raise them. When set, `env.j2` writes them into each server's `.env`, Compose passes them into the container, and `k3s/node-entrypoint.sh` adds `--etcd-arg=heartbeat-interval=<ms>` and `--etcd-arg=election-timeout=<ms>` to `k3s server`, on the mesh and when a self-hosted bootstrap server starts off it. Agents run no etcd and take neither.
+
+```yaml
+# group_vars/github_runner_cluster.yml
+github_runner_cluster_etcd_heartbeat_interval: 500
+github_runner_cluster_etcd_election_timeout: 5000
+```
+
+The role checks, before it changes anything on a host, that both are positive whole numbers, that they are set together or not at all, that the election timeout is at least five times the heartbeat interval (etcd refuses to start otherwise), and that every server of the group has the same two values, read from the inventory. etcd requires identical timing on every member, so set them once for the whole group.
+
+A change reaches a server when the role recreates its container, which restarts k3s there. Edit the inventory once, then either run the role against every server together, which restarts all the control-plane nodes at about the same time and so takes the API down until a majority is back, or roll it out one server at a time with `--limit <server>`, waiting for that node to be Ready before the next (the identical-values check reads the whole group's inventory, so a limited run passes). Members briefly disagree either way. The role cannot see the timing the running servers use, so it does not warn when they differ: after a rollout that stopped partway, compare each server's `/var/lib/rancher/k3s/server/db/etcd/config`, which holds `heartbeat-interval` and `election-timeout` once they were passed.
+
+An `--etcd-arg` is global: k3s also gives it to the temporary etcd it starts on every restart against an already-initialised datastore, never on a first boot. An earlier `listen-peer-urls` override broke exactly that restart, and only that one (see the first of the known issues in the [repository README](../../README.md) about `--etcd-arg`). These two arguments do not change what etcd binds to, but test a restart of an already-initialised datastore with them set before relying on them: `tests/mesh/run.sh` does that for the `hosted` scenario, restarting one server after the cluster is up. A timing that is too slow for the relay also lengthens how long a real leader failure takes to notice, so raise them only as far as the observed round trips need.
+
 ## Multiple sites
 
 Each site is its own cluster: its own inventory group, bootstrap server and mesh provider, run by its own play with `github_runner_cluster_group` set to that group (see `examples/two-site/`). One cluster spanning two meshes is not supported: flannel's tailscale backend carries pod traffic over the single mesh every node belongs to, so every node of a cluster must be on the same tailnet or Headscale server. A host belongs to at most one cluster group.
@@ -160,6 +176,7 @@ Each site is its own cluster: its own inventory group, bootstrap server and mesh
 - `github_runner_cluster_k3s_version`, `github_runner_cluster_tailscale_version`: the pinned image tags every node is built from, `github_runner_cluster_allow_k3s_version_jump` (default false) to skip the comparison with what runs, and `github_runner_cluster_cluster_read_timeout` (default 30, a positive whole number of seconds) for each server's read (see [Versions](#versions)). `github_runner_cluster_k3s_tested_version` is the k3s default, which the minor-version check compares hosts that set nothing against; set `github_runner_cluster_k3s_version` instead.
 - `github_runner_cluster_node_name`: the k3s node name and mesh hostname, defaulting to the inventory name.
 - `github_runner_cluster_node_taints`: Kubernetes taints (`key=value:Effect`, no spaces) the host's node registers with, empty by default. `NoSchedule` keeps pods without a matching toleration off it and `NoExecute` also evicts the ones already there. k3s applies them on every registration, so they come back if the node object is deleted, which a taint set with `kubectl` does not.
+- `github_runner_cluster_etcd_heartbeat_interval`, `github_runner_cluster_etcd_election_timeout`: etcd's timing in milliseconds for every server, empty by default (k3s's own). Set together, the same on every server, the election timeout at least five times the heartbeat interval; see [etcd timing](#etcd-timing).
 - Per-host overrides: `github_runner_cluster_node_role` (`server` or `agent`), `github_runner_cluster_bootstrap` (true picks the bootstrap host, false excludes a host), `github_runner_cluster_server_url` (join this server instead, for a cluster whose servers are outside the inventory), `github_runner_cluster_tls_sans`, `github_runner_cluster_node_address`. The run fails if the result has an even number of servers, no servers, or more than one bootstrap host.
 - `github_runner_cluster_docker_context` (default empty): pins the Docker context this host's node runs in (see [Docker contexts](#docker-contexts)).
 - `github_runner_cluster_runtime_autostart` (default true), `github_runner_cluster_runtime` (default `auto`; `colima`, `docker_desktop` or `other`), `github_runner_cluster_colima_profile` (default empty, taken from the Docker context), `github_runner_cluster_runtime_path` (default both Homebrew prefixes and the system directories): on a macOS host, start Colima at login through a launchd agent and check the runtime (see [Container runtime at login](#container-runtime-at-login-macos)).
@@ -204,6 +221,6 @@ or read from 1Password by adding `exadev.github_runner.github_runner_secrets_one
 
 `tests/litestream/run.sh` runs the `headscale_hosted` provider with Litestream and a warm standby against an S3-compatible store on this machine's Docker, registers a real Tailscale client, promotes the standby and checks that users, nodes and pre-auth keys survive, that the client reconnects and that the promoted server keeps replicating. It also checks that `headscale_in_cluster` refuses to run without Litestream. `.github/workflows/litestream.yml` runs it on pull requests that touch the Headscale providers.
 
-`tests/mesh/run.sh` brings up three k3s nodes in Docker on one machine and joins them through Headscale with this role, for the `headscale_hosted` and `headscale_in_cluster` providers, and checks the `headscale_existing` policy check against a real server. Each cluster scenario runs the role a second time and fails if any node container restarted. The `compose_faulty` scenario checks the role refuses a Compose release that has that fault. With `GRTEST_PIN_CONTEXT=1` the cluster scenarios pin `github_runner_cluster_docker_context` while the current Docker context the playbooks see points at a socket that does not exist, so any docker call that ignored the pin fails, and they check the current context is unchanged afterwards; `tests/litestream/run.sh` takes the same setting for the Headscale hosts it delegates to and the promotion playbook. The `context_checks` scenario checks the role refuses a pinned context that does not exist, and one set alongside `DOCKER_HOST`, before writing anything. `.github/workflows/mesh-integration.yml` runs them on pull requests that touch the role, the hosted scenario on 2.39.0, the first fixed release, as well as on a current one.
+`tests/mesh/run.sh` brings up three k3s nodes in Docker on one machine and joins them through Headscale with this role, for the `headscale_hosted` and `headscale_in_cluster` providers, and checks the `headscale_existing` policy check against a real server. Each cluster scenario runs the role a second time and fails if any node container restarted. Both set the etcd timing and check each server's etcd config file carries it, and the `hosted` scenario then restarts a joined server against its initialised datastore and checks the cluster recovers with the timing still in place. The `compose_faulty` scenario checks the role refuses a Compose release that has that fault. With `GRTEST_PIN_CONTEXT=1` the cluster scenarios pin `github_runner_cluster_docker_context` while the current Docker context the playbooks see points at a socket that does not exist, so any docker call that ignored the pin fails, and they check the current context is unchanged afterwards; `tests/litestream/run.sh` takes the same setting for the Headscale hosts it delegates to and the promotion playbook. The `context_checks` scenario checks the role refuses a pinned context that does not exist, and one set alongside `DOCKER_HOST`, before writing anything. `.github/workflows/mesh-integration.yml` runs them on pull requests that touch the role, the hosted scenario on 2.39.0, the first fixed release, as well as on a current one.
 
 `tests/unit/test_runtime_autostart.py` covers the [container runtime tasks](#container-runtime-at-login-macos) without a Mac: it renders the launchd agent through `tests/runtime_autostart/render_plist.yml`, and runs `tasks/runtime.yml` and `tasks/end_checks.yml` through the role against stand-ins for `docker`, `colima`, `launchctl` and `plutil` on `PATH`. It checks the variable defaults, that Linux hosts are left alone, that a second run changes nothing, that `--check` writes and starts nothing, and that each way the runtime or the node's container can be down fails with its message. `tests/unit/test_container_runtime.py` covers the filter that works out the runtime from the Docker context. Nothing there touches a real Colima, launchd or Docker Desktop, so loading the agent into a real login session at a real reboot is the part only a run on a Mac can show.

roles/github_runner_cluster/defaults/main.yml

@@ -21,6 +21,10 @@ github_runner_cluster_effective_node_name: "{{ github_runner_cluster_node_name i
 # Kubernetes taints this host's node registers with, as key=value:Effect strings (for example github-runner.example.com/excluded=true:NoExecute), none containing a space. A taint keeps pods without a matching toleration off the node; NoExecute also evicts the ones already there. They are applied by k3s each time the node registers, so they survive the node object being deleted and registering again, which a taint put on the node by hand does not. Per host, in its host_vars.
 github_runner_cluster_node_taints: []
 
+# etcd's heartbeat interval and election timeout, in milliseconds, for every k3s server (passed to k3s as --etcd-arg=heartbeat-interval=<ms> and --etcd-arg=election-timeout=<ms>). Empty, the default, leaves k3s's own timing in place (a 100 ms heartbeat and a 1000 ms election timeout), so nothing changes for an existing fleet. Raise them where the servers' peer traffic goes through a relay, whose round trips and jitter exceed what the defaults tolerate. Both are positive whole numbers, set together or not at all, the election timeout at least five times the heartbeat interval (etcd's own rule), and the same on every server of the group (tasks/etcd_timing.yml checks all three before the role changes anything). Agents run no etcd and ignore them.
+github_runner_cluster_etcd_heartbeat_interval: ""
+github_runner_cluster_etcd_election_timeout: ""
+
 # The images every node's k3s container is built from: the rancher/k3s tag (k3s's version with "+" written as "-") and the tailscale/tailscale tag whose tailscale and tailscaled binaries are copied in. Pinned, never latest: Compose builds a node's image only when it is missing, so a floating tag would put whatever version is newest on the day a node is first built, rebuilt or moved into a running cluster, unannounced. Set both once for the whole cluster group, in inventory or group_vars (the check below reads every host's through hostvars). Before it changes anything on a host the role fails if either is not a pinned tag, if the group's hosts name different k3s minor versions, or if the k3s version would break k3s's upgrade path or Kubernetes' version skew policy from what this node and the cluster run now (see the role README's Versions section). Changing them changes no running node: its image is kept until it is rebuilt (see the role README's Versions section for upgrading).
 github_runner_cluster_k3s_version: "{{ github_runner_cluster_k3s_tested_version }}"
 github_runner_cluster_tailscale_version: "v1.102.4"

roles/github_runner_cluster/files/docker-compose.yml

@@ -51,6 +51,9 @@ services:
       # Empty (not :?required) on the bootstrap host - the one server with nothing to join. Every OTHER server host's own .env sets this to the bootstrap host's own URL.
       - K3S_JOIN_SERVER_URL=${K3S_JOIN_SERVER_URL:-}
       - K3S_NODE_TAINTS=${K3S_NODE_TAINTS:-}
+      # Both empty (k3s's own etcd timing) or both set, in milliseconds; the Ansible role checks them before templating .env. The agent has no etcd and takes neither.
+      - K3S_ETCD_HEARTBEAT_INTERVAL=${K3S_ETCD_HEARTBEAT_INTERVAL:-}
+      - K3S_ETCD_ELECTION_TIMEOUT=${K3S_ETCD_ELECTION_TIMEOUT:-}
     volumes:
       - k3s-data:/var/lib/rancher/k3s
       # Persists this container's own Tailscale device identity/keys across recreations - without this, every recreate would register a brand new device on the tailnet instead of reconnecting as the same one, since the join key is reusable by design.

roles/github_runner_cluster/files/k3s/node-entrypoint.sh

@@ -1,7 +1,7 @@
 #!/bin/sh
 # Entry point for the k3s and k3s-agent services in docker-compose.yml: starts Tailscale inside the node's own container, then k3s with its --vpn-auth integration, so each node joins the mesh as its own device. Bind-mounted from the Compose project directory the role manages (github_runner_cluster_dir) rather than baked into the k3s image, so a host picks up a change to it with the Compose file that uses it, without rebuilding an image Compose would not otherwise rebuild.
 #
-# Usage: node-entrypoint.sh server|agent. Reads, from the container's environment (docker-compose.yml, from .env): K3S_VPN_AUTH_JOIN_KEY, K3S_VPN_AUTH_CONTROL_SERVER_URL (empty for Tailscale's own control plane, a Headscale URL otherwise), K3S_JOIN_SERVER_URL, K3S_TLS_SAN_LIST and K3S_DATASTORE (server), and K3S_MESH_SELF_HOSTED_HEALTH_URL (the bootstrap server of a cluster that runs its own Headscale, see supervise_self_hosted below).
+# Usage: node-entrypoint.sh server|agent. Reads, from the container's environment (docker-compose.yml, from .env): K3S_VPN_AUTH_JOIN_KEY, K3S_VPN_AUTH_CONTROL_SERVER_URL (empty for Tailscale's own control plane, a Headscale URL otherwise), K3S_JOIN_SERVER_URL, K3S_TLS_SAN_LIST, K3S_DATASTORE, K3S_ETCD_HEARTBEAT_INTERVAL and K3S_ETCD_ELECTION_TIMEOUT (server), and K3S_MESH_SELF_HOSTED_HEALTH_URL (the bootstrap server of a cluster that runs its own Headscale, see supervise_self_hosted below).
 # No set -e: a failed `tailscale up` falls through to waiting for an address, as it always has, rather than exiting and restarting the container.
 set -u
 
@@ -16,6 +16,18 @@ for node_taint in ${K3S_NODE_TAINTS:-}; do
   node_taint_flags="$node_taint_flags --node-taint $node_taint"
 done
 
+# Sets etcd_timing_flags from etcd's timing, for the server only (the agent runs no etcd): K3S_ETCD_HEARTBEAT_INTERVAL and K3S_ETCD_ELECTION_TIMEOUT, in milliseconds, both set or both empty (the role checks that the election timeout is at least five times the heartbeat interval, and that every server has the same values). Empty leaves etcd on k3s's own timing. Each is passed as an --etcd-arg, which k3s also hands to the temporary etcd it starts on every restart to reconcile with an initialised datastore; these two flags do not change what that instance binds to, unlike the listen-peer-urls override that once broke it (see the README's known issues).
+set_etcd_timing_flags() {
+  etcd_timing_flags=""
+  if [ -n "${K3S_ETCD_HEARTBEAT_INTERVAL:-}" ]; then
+    etcd_timing_flags="$etcd_timing_flags --etcd-arg=heartbeat-interval=$K3S_ETCD_HEARTBEAT_INTERVAL"
+  fi
+  if [ -n "${K3S_ETCD_ELECTION_TIMEOUT:-}" ]; then
+    etcd_timing_flags="$etcd_timing_flags --etcd-arg=election-timeout=$K3S_ETCD_ELECTION_TIMEOUT"
+  fi
+}
+set_etcd_timing_flags
+
 # k3s's own vpn-auth integration only runs `tailscale up`; it does not start the daemon itself (without this, k3s fails outright with "tailscale up failed: ... it doesn't appear to be running"). Waits for the control socket before anything talks to it.
 mkdir -p /var/run/tailscale /var/lib/tailscale
 tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
@@ -83,6 +95,8 @@ wait_for_join_server() {
 # --cluster-init (only when K3S_JOIN_SERVER_URL is unset, and K3S_DATASTORE is not sqlite) bootstraps embedded etcd, or converts an existing single-node SQLite datastore to it, the one migration k3s documents; once initialised, k3s ignores it on restart, so it stays in place permanently. A host with K3S_JOIN_SERVER_URL set passes --server <url> instead and joins as another control-plane/etcd member.
 #
 # --tls-san is added once per entry of K3S_TLS_SAN_LIST: every server's certificate must cover every server's own mesh name, since a client reaching any server directly by that name needs the certificate to match it.
+#
+# --etcd-arg=heartbeat-interval and --etcd-arg=election-timeout are added when set (set_etcd_timing_flags above), for peers whose traffic goes through a relay and so exceeds etcd's default timing.
 mesh_server_args() {
   set -- --disable=traefik --disable-network-policy --write-kubeconfig-mode=6443 \
     --vpn-auth="$vpn_auth" \
@@ -94,13 +108,15 @@ mesh_server_args() {
     set -- "$@" --cluster-init
   fi
   for san in ${K3S_TLS_SAN_LIST:-}; do set -- "$@" --tls-san "$san"; done
+  for flag in $etcd_timing_flags; do set -- "$@" "$flag"; done
   server_args="$*"
 }
 
 # Sets server_args to the arguments of `k3s server` off the mesh, for a self-hosted bootstrap server whose Headscale server is not answering yet. No --vpn-auth, no --node-ip and no mesh resolver, so k3s and the Headscale static pod come up on the container's own network. No --cluster-init either: a fresh server starts on SQLite, and restarting it on the mesh with --cluster-init converts that to etcd with the mesh IP as etcd's peer address from the start, while an already-initialised server finds its etcd datastore regardless.
 direct_server_args() {
   set -- --disable=traefik --disable-network-policy --write-kubeconfig-mode=6443
   for san in ${K3S_TLS_SAN_LIST:-}; do set -- "$@" --tls-san "$san"; done
+  for flag in $etcd_timing_flags; do set -- "$@" "$flag"; done
   server_args="$*"
 }