diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index 89cac7e..1e5e60a 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -143,6 +143,22 @@ The limitation: the control server lives on one node. While the bootstrap server Headscale's database lives on the bootstrap server's k3s data volume, and a static pod cannot use a replicated persistent volume instead, since kubelet runs it with no API server to bind a claim. So the provider requires Litestream: set `github_runner_cluster_headscale_litestream_replica` (see [Backups and a warm standby](#backups-and-a-warm-standby) for the settings), and the pod gets a `litestream-restore` init container and a `litestream` sidecar. A rebuilt bootstrap server then restores the database before Headscale starts, with the same data loss as `headscale_hosted`. The Noise key copy in `keys/` sits on the host, outside the k3s volume. It survives losing that volume but not the host, so supply `github_runner_cluster_headscale_noise_private_key` too if the host itself may be replaced. To run without Litestream, set `github_runner_cluster_headscale_accept_node_local_state: true`, and accept that losing that volume loses every registration: every node must then join again with a new key. There is no standby for this provider; recovery is rebuilding the bootstrap server. +## etcd timing + +k3s's embedded etcd runs with a 100 ms heartbeat interval and a 1000 ms election timeout. Where the servers' peer traffic goes through a relay, as it does when each node's container sits behind NAT, round trips and jitter can exceed that: followers miss heartbeats, elections repeat, reads against etcd fail as Unavailable and k3s exits and restarts, which kills the pods on that node. `github_runner_cluster_etcd_heartbeat_interval` and `github_runner_cluster_etcd_election_timeout` (milliseconds, both empty by default so k3s's own timing applies and nothing changes for an existing fleet) raise them. When set, `env.j2` writes them into each server's `.env`, Compose passes them into the container, and `k3s/node-entrypoint.sh` adds `--etcd-arg=heartbeat-interval=` and `--etcd-arg=election-timeout=` to `k3s server`, on the mesh and when a self-hosted bootstrap server starts off it. Agents run no etcd and take neither. + +```yaml +# group_vars/github_runner_cluster.yml +github_runner_cluster_etcd_heartbeat_interval: 500 +github_runner_cluster_etcd_election_timeout: 5000 +``` + +The role checks, before it changes anything on a host, that both are positive whole numbers, that they are set together or not at all, that the election timeout is at least five times the heartbeat interval (etcd refuses to start otherwise), and that every server of the group has the same two values, read from the inventory. etcd requires identical timing on every member, so set them once for the whole group. + +A change reaches a server when the role recreates its container, which restarts k3s there. Edit the inventory once, then either run the role against every server together, which restarts all the control-plane nodes at about the same time and so takes the API down until a majority is back, or roll it out one server at a time with `--limit `, waiting for that node to be Ready before the next (the identical-values check reads the whole group's inventory, so a limited run passes). Members briefly disagree either way. The role cannot see the timing the running servers use, so it does not warn when they differ: after a rollout that stopped partway, compare each server's `/var/lib/rancher/k3s/server/db/etcd/config`, which holds `heartbeat-interval` and `election-timeout` once they were passed. + +An `--etcd-arg` is global: k3s also gives it to the temporary etcd it starts on every restart against an already-initialised datastore, never on a first boot. An earlier `listen-peer-urls` override broke exactly that restart, and only that one (see the first of the known issues in the [repository README](../../README.md) about `--etcd-arg`). These two arguments do not change what etcd binds to, but test a restart of an already-initialised datastore with them set before relying on them: `tests/mesh/run.sh` does that for the `hosted` scenario, restarting one server after the cluster is up. A timing that is too slow for the relay also lengthens how long a real leader failure takes to notice, so raise them only as far as the observed round trips need. + ## Multiple sites Each site is its own cluster: its own inventory group, bootstrap server and mesh provider, run by its own play with `github_runner_cluster_group` set to that group (see `examples/two-site/`). One cluster spanning two meshes is not supported: flannel's tailscale backend carries pod traffic over the single mesh every node belongs to, so every node of a cluster must be on the same tailnet or Headscale server. A host belongs to at most one cluster group. @@ -160,6 +176,7 @@ Each site is its own cluster: its own inventory group, bootstrap server and mesh - `github_runner_cluster_k3s_version`, `github_runner_cluster_tailscale_version`: the pinned image tags every node is built from, `github_runner_cluster_allow_k3s_version_jump` (default false) to skip the comparison with what runs, and `github_runner_cluster_cluster_read_timeout` (default 30, a positive whole number of seconds) for each server's read (see [Versions](#versions)). `github_runner_cluster_k3s_tested_version` is the k3s default, which the minor-version check compares hosts that set nothing against; set `github_runner_cluster_k3s_version` instead. - `github_runner_cluster_node_name`: the k3s node name and mesh hostname, defaulting to the inventory name. - `github_runner_cluster_node_taints`: Kubernetes taints (`key=value:Effect`, no spaces) the host's node registers with, empty by default. `NoSchedule` keeps pods without a matching toleration off it and `NoExecute` also evicts the ones already there. k3s applies them on every registration, so they come back if the node object is deleted, which a taint set with `kubectl` does not. +- `github_runner_cluster_etcd_heartbeat_interval`, `github_runner_cluster_etcd_election_timeout`: etcd's timing in milliseconds for every server, empty by default (k3s's own). Set together, the same on every server, the election timeout at least five times the heartbeat interval; see [etcd timing](#etcd-timing). - Per-host overrides: `github_runner_cluster_node_role` (`server` or `agent`), `github_runner_cluster_bootstrap` (true picks the bootstrap host, false excludes a host), `github_runner_cluster_server_url` (join this server instead, for a cluster whose servers are outside the inventory), `github_runner_cluster_tls_sans`, `github_runner_cluster_node_address`. The run fails if the result has an even number of servers, no servers, or more than one bootstrap host. - `github_runner_cluster_docker_context` (default empty): pins the Docker context this host's node runs in (see [Docker contexts](#docker-contexts)). - `github_runner_cluster_runtime_autostart` (default true), `github_runner_cluster_runtime` (default `auto`; `colima`, `docker_desktop` or `other`), `github_runner_cluster_colima_profile` (default empty, taken from the Docker context), `github_runner_cluster_runtime_path` (default both Homebrew prefixes and the system directories): on a macOS host, start Colima at login through a launchd agent and check the runtime (see [Container runtime at login](#container-runtime-at-login-macos)). @@ -204,6 +221,6 @@ or read from 1Password by adding `exadev.github_runner.github_runner_secrets_one `tests/litestream/run.sh` runs the `headscale_hosted` provider with Litestream and a warm standby against an S3-compatible store on this machine's Docker, registers a real Tailscale client, promotes the standby and checks that users, nodes and pre-auth keys survive, that the client reconnects and that the promoted server keeps replicating. It also checks that `headscale_in_cluster` refuses to run without Litestream. `.github/workflows/litestream.yml` runs it on pull requests that touch the Headscale providers. -`tests/mesh/run.sh` brings up three k3s nodes in Docker on one machine and joins them through Headscale with this role, for the `headscale_hosted` and `headscale_in_cluster` providers, and checks the `headscale_existing` policy check against a real server. Each cluster scenario runs the role a second time and fails if any node container restarted. The `compose_faulty` scenario checks the role refuses a Compose release that has that fault. With `GRTEST_PIN_CONTEXT=1` the cluster scenarios pin `github_runner_cluster_docker_context` while the current Docker context the playbooks see points at a socket that does not exist, so any docker call that ignored the pin fails, and they check the current context is unchanged afterwards; `tests/litestream/run.sh` takes the same setting for the Headscale hosts it delegates to and the promotion playbook. The `context_checks` scenario checks the role refuses a pinned context that does not exist, and one set alongside `DOCKER_HOST`, before writing anything. `.github/workflows/mesh-integration.yml` runs them on pull requests that touch the role, the hosted scenario on 2.39.0, the first fixed release, as well as on a current one. +`tests/mesh/run.sh` brings up three k3s nodes in Docker on one machine and joins them through Headscale with this role, for the `headscale_hosted` and `headscale_in_cluster` providers, and checks the `headscale_existing` policy check against a real server. Each cluster scenario runs the role a second time and fails if any node container restarted. Both set the etcd timing and check each server's etcd config file carries it, and the `hosted` scenario then restarts a joined server against its initialised datastore and checks the cluster recovers with the timing still in place. The `compose_faulty` scenario checks the role refuses a Compose release that has that fault. With `GRTEST_PIN_CONTEXT=1` the cluster scenarios pin `github_runner_cluster_docker_context` while the current Docker context the playbooks see points at a socket that does not exist, so any docker call that ignored the pin fails, and they check the current context is unchanged afterwards; `tests/litestream/run.sh` takes the same setting for the Headscale hosts it delegates to and the promotion playbook. The `context_checks` scenario checks the role refuses a pinned context that does not exist, and one set alongside `DOCKER_HOST`, before writing anything. `.github/workflows/mesh-integration.yml` runs them on pull requests that touch the role, the hosted scenario on 2.39.0, the first fixed release, as well as on a current one. `tests/unit/test_runtime_autostart.py` covers the [container runtime tasks](#container-runtime-at-login-macos) without a Mac: it renders the launchd agent through `tests/runtime_autostart/render_plist.yml`, and runs `tasks/runtime.yml` and `tasks/end_checks.yml` through the role against stand-ins for `docker`, `colima`, `launchctl` and `plutil` on `PATH`. It checks the variable defaults, that Linux hosts are left alone, that a second run changes nothing, that `--check` writes and starts nothing, and that each way the runtime or the node's container can be down fails with its message. `tests/unit/test_container_runtime.py` covers the filter that works out the runtime from the Docker context. Nothing there touches a real Colima, launchd or Docker Desktop, so loading the agent into a real login session at a real reboot is the part only a run on a Mac can show. diff --git a/roles/github_runner_cluster/defaults/main.yml b/roles/github_runner_cluster/defaults/main.yml index 196952e..6a72c50 100644 --- a/roles/github_runner_cluster/defaults/main.yml +++ b/roles/github_runner_cluster/defaults/main.yml @@ -21,6 +21,10 @@ github_runner_cluster_effective_node_name: "{{ github_runner_cluster_node_name i # Kubernetes taints this host's node registers with, as key=value:Effect strings (for example github-runner.example.com/excluded=true:NoExecute), none containing a space. A taint keeps pods without a matching toleration off the node; NoExecute also evicts the ones already there. They are applied by k3s each time the node registers, so they survive the node object being deleted and registering again, which a taint put on the node by hand does not. Per host, in its host_vars. github_runner_cluster_node_taints: [] +# etcd's heartbeat interval and election timeout, in milliseconds, for every k3s server (passed to k3s as --etcd-arg=heartbeat-interval= and --etcd-arg=election-timeout=). Empty, the default, leaves k3s's own timing in place (a 100 ms heartbeat and a 1000 ms election timeout), so nothing changes for an existing fleet. Raise them where the servers' peer traffic goes through a relay, whose round trips and jitter exceed what the defaults tolerate. Both are positive whole numbers, set together or not at all, the election timeout at least five times the heartbeat interval (etcd's own rule), and the same on every server of the group (tasks/etcd_timing.yml checks all three before the role changes anything). Agents run no etcd and ignore them. +github_runner_cluster_etcd_heartbeat_interval: "" +github_runner_cluster_etcd_election_timeout: "" + # The images every node's k3s container is built from: the rancher/k3s tag (k3s's version with "+" written as "-") and the tailscale/tailscale tag whose tailscale and tailscaled binaries are copied in. Pinned, never latest: Compose builds a node's image only when it is missing, so a floating tag would put whatever version is newest on the day a node is first built, rebuilt or moved into a running cluster, unannounced. Set both once for the whole cluster group, in inventory or group_vars (the check below reads every host's through hostvars). Before it changes anything on a host the role fails if either is not a pinned tag, if the group's hosts name different k3s minor versions, or if the k3s version would break k3s's upgrade path or Kubernetes' version skew policy from what this node and the cluster run now (see the role README's Versions section). Changing them changes no running node: its image is kept until it is rebuilt (see the role README's Versions section for upgrading). github_runner_cluster_k3s_version: "{{ github_runner_cluster_k3s_tested_version }}" github_runner_cluster_tailscale_version: "v1.102.4" diff --git a/roles/github_runner_cluster/files/docker-compose.yml b/roles/github_runner_cluster/files/docker-compose.yml index 6308d12..d0acb22 100644 --- a/roles/github_runner_cluster/files/docker-compose.yml +++ b/roles/github_runner_cluster/files/docker-compose.yml @@ -51,6 +51,9 @@ services: # Empty (not :?required) on the bootstrap host - the one server with nothing to join. Every OTHER server host's own .env sets this to the bootstrap host's own URL. - K3S_JOIN_SERVER_URL=${K3S_JOIN_SERVER_URL:-} - K3S_NODE_TAINTS=${K3S_NODE_TAINTS:-} + # Both empty (k3s's own etcd timing) or both set, in milliseconds; the Ansible role checks them before templating .env. The agent has no etcd and takes neither. + - K3S_ETCD_HEARTBEAT_INTERVAL=${K3S_ETCD_HEARTBEAT_INTERVAL:-} + - K3S_ETCD_ELECTION_TIMEOUT=${K3S_ETCD_ELECTION_TIMEOUT:-} volumes: - k3s-data:/var/lib/rancher/k3s # Persists this container's own Tailscale device identity/keys across recreations - without this, every recreate would register a brand new device on the tailnet instead of reconnecting as the same one, since the join key is reusable by design. diff --git a/roles/github_runner_cluster/files/k3s/node-entrypoint.sh b/roles/github_runner_cluster/files/k3s/node-entrypoint.sh index 4b1f5df..79febd6 100755 --- a/roles/github_runner_cluster/files/k3s/node-entrypoint.sh +++ b/roles/github_runner_cluster/files/k3s/node-entrypoint.sh @@ -1,7 +1,7 @@ #!/bin/sh # Entry point for the k3s and k3s-agent services in docker-compose.yml: starts Tailscale inside the node's own container, then k3s with its --vpn-auth integration, so each node joins the mesh as its own device. Bind-mounted from the Compose project directory the role manages (github_runner_cluster_dir) rather than baked into the k3s image, so a host picks up a change to it with the Compose file that uses it, without rebuilding an image Compose would not otherwise rebuild. # -# Usage: node-entrypoint.sh server|agent. Reads, from the container's environment (docker-compose.yml, from .env): K3S_VPN_AUTH_JOIN_KEY, K3S_VPN_AUTH_CONTROL_SERVER_URL (empty for Tailscale's own control plane, a Headscale URL otherwise), K3S_JOIN_SERVER_URL, K3S_TLS_SAN_LIST and K3S_DATASTORE (server), and K3S_MESH_SELF_HOSTED_HEALTH_URL (the bootstrap server of a cluster that runs its own Headscale, see supervise_self_hosted below). +# Usage: node-entrypoint.sh server|agent. Reads, from the container's environment (docker-compose.yml, from .env): K3S_VPN_AUTH_JOIN_KEY, K3S_VPN_AUTH_CONTROL_SERVER_URL (empty for Tailscale's own control plane, a Headscale URL otherwise), K3S_JOIN_SERVER_URL, K3S_TLS_SAN_LIST, K3S_DATASTORE, K3S_ETCD_HEARTBEAT_INTERVAL and K3S_ETCD_ELECTION_TIMEOUT (server), and K3S_MESH_SELF_HOSTED_HEALTH_URL (the bootstrap server of a cluster that runs its own Headscale, see supervise_self_hosted below). # No set -e: a failed `tailscale up` falls through to waiting for an address, as it always has, rather than exiting and restarting the container. set -u @@ -16,6 +16,18 @@ for node_taint in ${K3S_NODE_TAINTS:-}; do node_taint_flags="$node_taint_flags --node-taint $node_taint" done +# Sets etcd_timing_flags from etcd's timing, for the server only (the agent runs no etcd): K3S_ETCD_HEARTBEAT_INTERVAL and K3S_ETCD_ELECTION_TIMEOUT, in milliseconds, both set or both empty (the role checks that the election timeout is at least five times the heartbeat interval, and that every server has the same values). Empty leaves etcd on k3s's own timing. Each is passed as an --etcd-arg, which k3s also hands to the temporary etcd it starts on every restart to reconcile with an initialised datastore; these two flags do not change what that instance binds to, unlike the listen-peer-urls override that once broke it (see the README's known issues). +set_etcd_timing_flags() { + etcd_timing_flags="" + if [ -n "${K3S_ETCD_HEARTBEAT_INTERVAL:-}" ]; then + etcd_timing_flags="$etcd_timing_flags --etcd-arg=heartbeat-interval=$K3S_ETCD_HEARTBEAT_INTERVAL" + fi + if [ -n "${K3S_ETCD_ELECTION_TIMEOUT:-}" ]; then + etcd_timing_flags="$etcd_timing_flags --etcd-arg=election-timeout=$K3S_ETCD_ELECTION_TIMEOUT" + fi +} +set_etcd_timing_flags + # k3s's own vpn-auth integration only runs `tailscale up`; it does not start the daemon itself (without this, k3s fails outright with "tailscale up failed: ... it doesn't appear to be running"). Waits for the control socket before anything talks to it. mkdir -p /var/run/tailscale /var/lib/tailscale tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock & @@ -83,6 +95,8 @@ wait_for_join_server() { # --cluster-init (only when K3S_JOIN_SERVER_URL is unset, and K3S_DATASTORE is not sqlite) bootstraps embedded etcd, or converts an existing single-node SQLite datastore to it, the one migration k3s documents; once initialised, k3s ignores it on restart, so it stays in place permanently. A host with K3S_JOIN_SERVER_URL set passes --server instead and joins as another control-plane/etcd member. # # --tls-san is added once per entry of K3S_TLS_SAN_LIST: every server's certificate must cover every server's own mesh name, since a client reaching any server directly by that name needs the certificate to match it. +# +# --etcd-arg=heartbeat-interval and --etcd-arg=election-timeout are added when set (set_etcd_timing_flags above), for peers whose traffic goes through a relay and so exceeds etcd's default timing. mesh_server_args() { set -- --disable=traefik --disable-network-policy --write-kubeconfig-mode=6443 \ --vpn-auth="$vpn_auth" \ @@ -94,6 +108,7 @@ mesh_server_args() { set -- "$@" --cluster-init fi for san in ${K3S_TLS_SAN_LIST:-}; do set -- "$@" --tls-san "$san"; done + for flag in $etcd_timing_flags; do set -- "$@" "$flag"; done server_args="$*" } @@ -101,6 +116,7 @@ mesh_server_args() { direct_server_args() { set -- --disable=traefik --disable-network-policy --write-kubeconfig-mode=6443 for san in ${K3S_TLS_SAN_LIST:-}; do set -- "$@" --tls-san "$san"; done + for flag in $etcd_timing_flags; do set -- "$@" "$flag"; done server_args="$*" } diff --git a/roles/github_runner_cluster/tasks/etcd_timing.yml b/roles/github_runner_cluster/tasks/etcd_timing.yml new file mode 100644 index 0000000..e2dfe22 --- /dev/null +++ b/roles/github_runner_cluster/tasks/etcd_timing.yml @@ -0,0 +1,54 @@ +--- +# Checks etcd's heartbeat interval and election timeout (github_runner_cluster_etcd_heartbeat_interval, github_runner_cluster_etcd_election_timeout) before the role changes anything on the host. Run after the topology is known, since the servers must agree. Both unset is the default and passes: k3s then keeps its own timing. + +- name: Fail if only one of etcd's heartbeat interval and election timeout is set + # etcd requires the election timeout to be at least five times the heartbeat interval, which cannot be judged against the other value's k3s default without knowing it, so the two are set together. + ansible.builtin.assert: + that: (github_runner_cluster_etcd_heartbeat_interval | string | length > 0) == (github_runner_cluster_etcd_election_timeout | string | length > 0) + quiet: true + fail_msg: >- + github_runner_cluster_etcd_heartbeat_interval is '{{ github_runner_cluster_etcd_heartbeat_interval }}' and + github_runner_cluster_etcd_election_timeout is '{{ github_runner_cluster_etcd_election_timeout }}'; set both or neither. + Nothing was changed. + +- name: Fail if etcd's heartbeat interval or election timeout is not a positive whole number of milliseconds + ansible.builtin.assert: + that: + - (github_runner_cluster_etcd_heartbeat_interval | string) is match('^[1-9][0-9]*\\Z') + - (github_runner_cluster_etcd_election_timeout | string) is match('^[1-9][0-9]*\\Z') + quiet: true + fail_msg: >- + github_runner_cluster_etcd_heartbeat_interval is '{{ github_runner_cluster_etcd_heartbeat_interval }}' and + github_runner_cluster_etcd_election_timeout is '{{ github_runner_cluster_etcd_election_timeout }}'; each must be a + positive whole number of milliseconds. Nothing was changed. + when: github_runner_cluster_etcd_heartbeat_interval | string | length > 0 + +- name: Fail if etcd's election timeout is under five times its heartbeat interval + ansible.builtin.assert: + that: (github_runner_cluster_etcd_election_timeout | int) >= 5 * (github_runner_cluster_etcd_heartbeat_interval | int) + quiet: true + fail_msg: >- + github_runner_cluster_etcd_election_timeout ({{ github_runner_cluster_etcd_election_timeout }} ms) must be at least five + times github_runner_cluster_etcd_heartbeat_interval ({{ github_runner_cluster_etcd_heartbeat_interval }} ms), which is + {{ 5 * (github_runner_cluster_etcd_heartbeat_interval | int) }} ms; etcd refuses to start with a shorter one. Nothing was changed. + when: github_runner_cluster_etcd_heartbeat_interval | string | length > 0 + +- name: Fail if the cluster's servers are given different etcd timing + # Every host is compared through hostvars, its own included, so the result is the same on whichever host runs it, and under --limit to the host that is out of step. etcd members that disagree on the heartbeat interval or election timeout elect and fail over unreliably. Changing the timing means changing it for the whole group and applying it to every server (see the role README). + vars: + github_runner_cluster_etcd_timing_by_server: >- + {%- set by_server = {} -%} + {%- for server in github_runner_cluster_topology.servers -%} + {%- set heartbeat = hostvars[server].github_runner_cluster_etcd_heartbeat_interval | default('') | string -%} + {%- set election = hostvars[server].github_runner_cluster_etcd_election_timeout | default('') | string -%} + {%- set _ = by_server.update({server: (heartbeat ~ ' ms heartbeat, ' ~ election ~ ' ms election') if (heartbeat ~ election) | length > 0 else "k3s's own"}) -%} + {%- endfor -%} + {{ by_server }} + ansible.builtin.assert: + that: github_runner_cluster_etcd_timing_by_server.values() | unique | list | length <= 1 + quiet: true + fail_msg: >- + The servers of {{ github_runner_cluster_group }} do not all have the same etcd timing + ({% for server, timing in github_runner_cluster_etcd_timing_by_server.items() %}{{ server }}: {{ timing }}{{ '; ' if not loop.last }}{% endfor %}). + Set github_runner_cluster_etcd_heartbeat_interval and github_runner_cluster_etcd_election_timeout once for the whole group. + Nothing was changed. diff --git a/roles/github_runner_cluster/tasks/main.yml b/roles/github_runner_cluster/tasks/main.yml index 0c1f5a1..fb69b80 100644 --- a/roles/github_runner_cluster/tasks/main.yml +++ b/roles/github_runner_cluster/tasks/main.yml @@ -49,6 +49,9 @@ - name: Check the k3s version against what this node and the cluster run now ansible.builtin.include_tasks: running_version.yml +- name: Check etcd's heartbeat interval and election timeout + ansible.builtin.include_tasks: etcd_timing.yml + - name: Fail if the k3s token is not set ansible.builtin.fail: msg: >- diff --git a/roles/github_runner_cluster/templates/env.j2 b/roles/github_runner_cluster/templates/env.j2 index 71b7f47..66af1e0 100644 --- a/roles/github_runner_cluster/templates/env.j2 +++ b/roles/github_runner_cluster/templates/env.j2 @@ -18,3 +18,7 @@ K3S_NODE_NAME={{ github_runner_cluster_effective_node_name }} {% if github_runner_cluster_node_taints | length > 0 %} K3S_NODE_TAINTS={{ github_runner_cluster_node_taints | join(' ') }} {% endif %} +{% if github_runner_cluster_etcd_heartbeat_interval | string | length > 0 %} +K3S_ETCD_HEARTBEAT_INTERVAL={{ github_runner_cluster_etcd_heartbeat_interval }} +K3S_ETCD_ELECTION_TIMEOUT={{ github_runner_cluster_etcd_election_timeout }} +{% endif %} diff --git a/tests/mesh/run.sh b/tests/mesh/run.sh index 75370c6..cd56726 100755 --- a/tests/mesh/run.sh +++ b/tests/mesh/run.sh @@ -19,6 +19,9 @@ etcd_image=gcr.io/etcd-development/etcd:v3.5.21 busybox_image=busybox:1.37 # Tries per pod pair, 5 seconds apart plus each try's own timeout: about two minutes for the mesh to converge after a restart. reach_attempts=8 +# etcd's timing for every server, in milliseconds: a heartbeat interval and an election timeout of ten times it, above k3s's own defaults and so told apart from them, and above etcd's required five times. +etcd_heartbeat_ms=500 +etcd_election_ms=$((etcd_heartbeat_ms * 10)) work="${GRTEST_WORK:-$(mktemp -d "${TMPDIR:-/tmp}/grtest-mesh.XXXXXX")}" k3s_token="grtest-$(od -An -N12 -tx1 /dev/urandom | tr -d ' \n')" # The Docker CLI configuration the ansible-playbook runs see when the context is pinned; see tests/lib/docker_context.sh. @@ -161,6 +164,8 @@ EOF echo " github_runner_cluster_headscale_compose_files: [${work}/headscale-grtest.yml]" echo " github_runner_cluster_compose_files: [compose.grtest.yml]" echo " github_runner_cluster_k3s_token: ${k3s_token}" + echo " github_runner_cluster_etcd_heartbeat_interval: ${etcd_heartbeat_ms}" + echo " github_runner_cluster_etcd_election_timeout: ${etcd_election_ms}" if [ "$pin_context" = 1 ]; then echo " github_runner_cluster_docker_context: ${grtest_pinned_context}" fi @@ -258,6 +263,37 @@ EOF log "Pod-to-pod traffic crosses the mesh" } +# Every server's etcd must have been started with the timing the inventory sets: k3s writes the --etcd-arg values into the config file it starts etcd from. +assert_etcd_timing() { + local index config + for index in $(seq 1 "$node_count"); do + config=$(docker exec "$(container "$index")" cat /var/lib/rancher/k3s/server/db/etcd/config ". +echo_pod_state() { + kubectl_on 1 get pods -l app=grtest-echo --field-selector "spec.nodeName=$(node_name "$1")" \ + -o jsonpath='{.items[0].status.containerStatuses[0].restartCount} {.items[0].status.containerStatuses[0].ready}' 2>/dev/null +} + +# Restarts a node's container and returns once the cluster has seen it. The API keeps reporting the node and its pods as Ready until the node controller's grace period runs out, so waiting for Ready right after the restart passes at once on the old state, and the pod-to-pod checks then run before the restarted node's pod network exists again. The restart kills the containers of every pod on the node, so the node's echo pod is restarted by its kubelet: the wait is for that pod's restart count to rise and the pod to be ready again, and the checks after it read the pod addresses afresh. +restart_joined_server() { + local index="$1" before after + before=$(echo_pod_state "$index") || fail "cannot read the echo pod on $(node_name "$index") before restarting it" + docker restart "$(container "$index")" >/dev/null + local deadline=$((SECONDS + 600)) + while :; do + after=$(echo_pod_state "$index" || true) + if [ -n "$after" ] && [ "${after% *}" -gt "${before% *}" ] && [ "${after#* }" = true ]; then break; fi + [ "$SECONDS" -lt "$deadline" ] || { kubectl_on 1 get pods -l app=grtest-echo -o wide || true; fail "the echo pod on $(node_name "$index") did not restart and become ready again after its node restarted (was: ${before}, now: ${after:-unreadable})"; } + sleep 5 + done + log "$(node_name "$index") is back: its echo pod restarted (restart count ${before% *} -> ${after% *})" +} + # Whether a pod reaches another pod's echo server, allowing the mesh time to converge: after a node restarts, its peers take a while to learn its new endpoints and routes again. reaches() { local _ @@ -280,6 +316,8 @@ scenario_cluster() { write_inventory "$scenario" run_role assert_cluster + log "Checking every server's etcd runs with the timing the inventory sets" + assert_etcd_timing log "Rerunning the role, which must not restart any node" local before after before=$(started_at) @@ -287,6 +325,14 @@ scenario_cluster() { after=$(started_at) [ "$before" = "$after" ] || fail "rerunning the role restarted a node container: ${before} -> ${after}" assert_cluster + assert_etcd_timing + if [ "$scenario" = hosted ]; then + # A restart starts k3s's temporary etcd against the initialised datastore, which every --etcd-arg also reaches; the timing flags must not break it. The joined server is restarted, not the bootstrap one the Headscale server runs beside. + log "Restarting a joined server against its initialised datastore, with the etcd timing set" + restart_joined_server 2 + assert_cluster + assert_etcd_timing + fi if [ "$scenario" = in_cluster ]; then log "Cold-restarting the bootstrap server, which cannot rejoin a three-server cluster on its own, then recovering it" ansible_run -i "${work}/inventory.yml" "${repo_root}/playbooks/recover_in_cluster_mesh.yml" \ diff --git a/tests/unit/test_cluster_etcd_timing.py b/tests/unit/test_cluster_etcd_timing.py new file mode 100644 index 0000000..cdc7afd --- /dev/null +++ b/tests/unit/test_cluster_etcd_timing.py @@ -0,0 +1,234 @@ +"""Tests for the cluster role's etcd timing settings: the defaults, the checks in tasks/etcd_timing.yml (run through the role with tests/cluster_versions/checks.yml), the server's .env, the Compose file, and the flags node-entrypoint.sh passes to k3s.""" + +from __future__ import annotations + +import json +import os +import re +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path +from typing import Any + +import yaml + +ROOT = Path(__file__).resolve().parents[2] +ROLE = ROOT / "roles" / "github_runner_cluster" +ENTRYPOINT = ROLE / "files" / "k3s" / "node-entrypoint.sh" +COMPOSE = ROLE / "files" / "docker-compose.yml" +ANSIBLE_PLAYBOOK = os.environ.get("ANSIBLE_PLAYBOOK", "ansible-playbook") +DEFAULTS: dict[str, Any] = yaml.safe_load((ROLE / "defaults" / "main.yml").read_text()) + +HEARTBEAT = "github_runner_cluster_etcd_heartbeat_interval" +ELECTION = "github_runner_cluster_etcd_election_timeout" + + +def ansible_env() -> dict[str, str]: + return {**os.environ, "ANSIBLE_COLLECTIONS_PATH": str(ROOT / "playbooks" / "collections"), "ANSIBLE_LOCALHOST_WARNING": "false", "ANSIBLE_INVENTORY_UNPARSED_WARNING": "false", "ANSIBLE_RETRY_FILES_ENABLED": "false"} + + +class DefaultsTest(unittest.TestCase): + def test_both_are_unset_so_k3s_keeps_its_own_timing(self) -> None: + self.assertEqual(DEFAULTS[HEARTBEAT], "") + self.assertEqual(DEFAULTS[ELECTION], "") + + +@unittest.skipIf(shutil.which(ANSIBLE_PLAYBOOK) is None, "ansible-playbook is not installed") +class RoleTest(unittest.TestCase): + def setUp(self) -> None: + self.work = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, self.work) + + def check(self, hosts: dict[str, dict[str, Any]] | None = None, **variables: Any) -> subprocess.CompletedProcess[str]: + """Run the topology and etcd timing checks on a group of hosts (three servers unless ``hosts`` says otherwise), with ``variables`` as group variables.""" + inventory = self.work / "inventory.yml" + group = {"vars": variables, "hosts": hosts or {"node1": {}, "node2": {}, "node3": {}}} + inventory.write_text(yaml.safe_dump({"all": {"children": {"github_runner_cluster": group}}})) + args = [ANSIBLE_PLAYBOOK, "-i", str(inventory), str(ROOT / "tests" / "cluster_versions" / "checks.yml"), "-e", "check_tasks=topology,etcd_timing"] + return subprocess.run(args, capture_output=True, text=True, check=False, env=ansible_env(), cwd=self.work) + + def test_neither_set_passes(self) -> None: + result = self.check() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_a_valid_pair_passes(self) -> None: + for heartbeat, election in ((500, 5000), ("250", "1250"), (1, 5)): + with self.subTest(heartbeat=heartbeat, election=election): + result = self.check(**{HEARTBEAT: heartbeat, ELECTION: election}) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_an_election_timeout_under_five_times_the_heartbeat_is_refused(self) -> None: + for heartbeat, election in ((500, 2499), (1, 4)): + with self.subTest(heartbeat=heartbeat, election=election): + result = self.check(**{HEARTBEAT: heartbeat, ELECTION: election}) + self.assertNotEqual(result.returncode, 0) + self.assertIn(f"must be at least five times github_runner_cluster_etcd_heartbeat_interval ({heartbeat} ms), which is {5 * heartbeat} ms", result.stdout) + + def test_one_set_without_the_other_is_refused(self) -> None: + for variables in ({HEARTBEAT: 500}, {ELECTION: 5000}, {HEARTBEAT: 500, ELECTION: ""}): + with self.subTest(variables=variables): + result = self.check(**variables) + self.assertNotEqual(result.returncode, 0) + self.assertIn("set both or neither", result.stdout) + + def test_a_value_that_is_not_a_positive_whole_number_is_refused(self) -> None: + for value in ("abc", "1.5", -100, 0, "010", True): + with self.subTest(heartbeat=value): + result = self.check(**{HEARTBEAT: value, ELECTION: 100000}) + self.assertNotEqual(result.returncode, 0) + self.assertIn("must be a positive whole number of milliseconds", result.stdout) + with self.subTest(election=value): + result = self.check(**{HEARTBEAT: 100, ELECTION: value}) + self.assertNotEqual(result.returncode, 0) + self.assertIn("must be a positive whole number of milliseconds", result.stdout) + + def test_servers_with_different_timing_are_refused_on_every_host(self) -> None: + hosts = {"node1": {}, "node2": {HEARTBEAT: 500, ELECTION: 5000}, "node3": {}} + result = self.check(hosts) + self.assertNotEqual(result.returncode, 0) + self.assertIn("node1: k3s's own; node2: 500 ms heartbeat, 5000 ms election; node3: k3s's own", result.stdout.replace("\n", " ")) + self.assertEqual(sorted(re.findall(r"fatal: \[(node\d)\]", result.stdout)), ["node1", "node2", "node3"]) + + def test_servers_that_differ_in_one_value_are_refused(self) -> None: + hosts = {"node1": {HEARTBEAT: 500, ELECTION: 5000}, "node2": {HEARTBEAT: 500, ELECTION: 6000}, "node3": {HEARTBEAT: 500, ELECTION: 5000}} + result = self.check(hosts) + self.assertNotEqual(result.returncode, 0) + self.assertIn("do not all have the same etcd timing", result.stdout) + + def test_the_same_timing_set_per_host_passes(self) -> None: + result = self.check({name: {HEARTBEAT: 500, ELECTION: 5000} for name in ("node1", "node2", "node3")}) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_an_agent_with_other_timing_is_not_compared_with_the_servers(self) -> None: + # Agents run no etcd, so only the servers must agree; the fourth host is an agent (three servers are chosen automatically from four hosts). + hosts = {"node1": {HEARTBEAT: 500, ELECTION: 5000}, "node2": {HEARTBEAT: 500, ELECTION: 5000}, "node3": {HEARTBEAT: 500, ELECTION: 5000}, "node4": {}} + result = self.check(hosts) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def render_env(self, *extra: str) -> tuple[list[str], list[str]]: + env, agent = self.work / "env", self.work / "env-agent" + args = [ANSIBLE_PLAYBOOK, str(ROOT / "tests" / "cluster_versions" / "render_env.yml"), "-e", f"env_output={env}", "-e", f"env_agent_output={agent}", "-e", "github_runner_cluster_dir=/srv/github-runner", *extra] + result = subprocess.run(args, capture_output=True, text=True, check=False, env=ansible_env(), cwd=self.work) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + return env.read_text().splitlines(), agent.read_text().splitlines() + + def test_the_server_env_carries_both_values_when_set(self) -> None: + server, _ = self.render_env("-e", f"{HEARTBEAT}=500", "-e", f"{ELECTION}=5000") + self.assertIn("K3S_ETCD_HEARTBEAT_INTERVAL=500", server) + self.assertIn("K3S_ETCD_ELECTION_TIMEOUT=5000", server) + + def test_the_server_env_names_neither_by_default(self) -> None: + server, agent = self.render_env() + for lines in (server, agent): + self.assertEqual([line for line in lines if "ETCD" in line], []) + + def test_the_agent_env_never_carries_them(self) -> None: + _, agent = self.render_env("-e", f"{HEARTBEAT}=500", "-e", f"{ELECTION}=5000") + self.assertEqual([line for line in agent if "ETCD" in line], []) + + +class ComposeFileTest(unittest.TestCase): + def test_only_the_server_service_takes_them_and_empty_by_default(self) -> None: + services = yaml.safe_load(COMPOSE.read_text())["services"] + self.assertIn("K3S_ETCD_HEARTBEAT_INTERVAL=${K3S_ETCD_HEARTBEAT_INTERVAL:-}", services["k3s"]["environment"]) + self.assertIn("K3S_ETCD_ELECTION_TIMEOUT=${K3S_ETCD_ELECTION_TIMEOUT:-}", services["k3s"]["environment"]) + for name, service in services.items(): + if name != "k3s": + with self.subTest(service=name): + self.assertEqual([entry for entry in service.get("environment", []) if "ETCD" in entry], []) + + +def has_compose() -> bool: + if shutil.which("docker") is None: + return False + return subprocess.run(["docker", "compose", "version"], capture_output=True, check=False).returncode == 0 + + +@unittest.skipUnless(has_compose(), "docker compose is not installed") +class ComposeRenderTest(unittest.TestCase): + def environment(self, **env: str) -> dict[str, str]: + clean = {key: value for key, value in os.environ.items() if not key.startswith(("DOCKER_", "COMPOSE_", "K3S_", "TAILSCALE_"))} + args = ["docker", "compose", "--project-directory", tempfile.gettempdir(), "--project-name", "example", "-f", str(COMPOSE), "--profile", "server", "--profile", "agent", "config", "--format", "json"] + base = {"K3S_TOKEN": "t", "K3S_TLS_SAN_LIST": "n", "K3S_VERSION": "v1.34.1-k3s1", "TAILSCALE_VERSION": "v1.102.4"} + result = subprocess.run(args, capture_output=True, text=True, check=False, env=clean | base | env) + self.assertEqual(result.returncode, 0, result.stderr) + environment: dict[str, str] = json.loads(result.stdout)["services"]["k3s"]["environment"] + return environment + + def test_the_container_gets_the_values_from_the_env_file(self) -> None: + environment = self.environment(K3S_ETCD_HEARTBEAT_INTERVAL="500", K3S_ETCD_ELECTION_TIMEOUT="5000") + self.assertEqual(environment["K3S_ETCD_HEARTBEAT_INTERVAL"], "500") + self.assertEqual(environment["K3S_ETCD_ELECTION_TIMEOUT"], "5000") + + def test_the_container_gets_empty_values_when_the_env_file_has_none(self) -> None: + environment = self.environment() + self.assertEqual(environment["K3S_ETCD_HEARTBEAT_INTERVAL"], "") + self.assertEqual(environment["K3S_ETCD_ELECTION_TIMEOUT"], "") + + +def function_source(name: str) -> str: + """The definition of a shell function in the entrypoint, from its first line to the closing brace at the start of a line.""" + match = re.search(rf"^{name}\(\) \{{\n.*?^\}}\n", ENTRYPOINT.read_text(), re.MULTILINE | re.DOTALL) + assert match is not None, f"{name} is not defined in node-entrypoint.sh" + return match.group(0) + + +class EntrypointTest(unittest.TestCase): + """Runs the entrypoint's own argument-list functions in a shell, without Tailscale or k3s: the script starts tailscaled and execs k3s at top level, so only the functions and the call that sets the timing flags are taken from it.""" + + def server_args(self, builder: str, **env: str) -> list[str]: + script = "\n".join( + [ + "set -u", + 'vpn_auth="name=tailscale,joinKey=example"', + 'mesh_ip="100.64.0.1"', + function_source("set_etcd_timing_flags"), + function_source(builder), + "set_etcd_timing_flags", + f"{builder}", + 'printf "%s\\n" "$server_args"', + ] + ) + clean = {key: value for key, value in os.environ.items() if not key.startswith("K3S_")} + result = subprocess.run(["/bin/sh", "-c", script], capture_output=True, text=True, check=False, env=clean | env) + self.assertEqual(result.returncode, 0, result.stderr) + return result.stdout.split() + + def test_the_script_calls_the_function_that_sets_the_flags(self) -> None: + self.assertRegex(ENTRYPOINT.read_text(), re.compile(r"^set_etcd_timing_flags$", re.MULTILINE)) + + def test_both_flags_follow_the_other_server_arguments_on_the_mesh_and_off_it(self) -> None: + for builder in ("mesh_server_args", "direct_server_args"): + with self.subTest(builder=builder): + args = self.server_args(builder, K3S_ETCD_HEARTBEAT_INTERVAL="500", K3S_ETCD_ELECTION_TIMEOUT="5000", K3S_TLS_SAN_LIST="a b") + self.assertEqual(args[-2:], ["--etcd-arg=heartbeat-interval=500", "--etcd-arg=election-timeout=5000"]) + self.assertEqual(args.count("--tls-san"), 2) + + def test_a_joining_server_gets_them_too(self) -> None: + args = self.server_args("mesh_server_args", K3S_ETCD_HEARTBEAT_INTERVAL="500", K3S_ETCD_ELECTION_TIMEOUT="5000", K3S_JOIN_SERVER_URL="https://bootstrap.example.internal:6443") + self.assertIn("--server", args) + self.assertIn("--etcd-arg=heartbeat-interval=500", args) + self.assertIn("--etcd-arg=election-timeout=5000", args) + + def test_neither_variable_leaves_the_arguments_without_any_etcd_arg(self) -> None: + for builder in ("mesh_server_args", "direct_server_args"): + with self.subTest(builder=builder): + self.assertEqual([arg for arg in self.server_args(builder) if "etcd" in arg], []) + self.assertEqual([arg for arg in self.server_args(builder, K3S_ETCD_HEARTBEAT_INTERVAL="", K3S_ETCD_ELECTION_TIMEOUT="") if "etcd" in arg], []) + + def test_the_arguments_without_the_variables_are_what_they_were_before(self) -> None: + self.assertEqual(self.server_args("direct_server_args"), ["--disable=traefik", "--disable-network-policy", "--write-kubeconfig-mode=6443"]) + self.assertEqual( + self.server_args("mesh_server_args"), + ["--disable=traefik", "--disable-network-policy", "--write-kubeconfig-mode=6443", "--vpn-auth=name=tailscale,joinKey=example", "--resolv-conf=/etc/k3s-resolv.conf", "--node-ip=100.64.0.1", "--cluster-init"], + ) + + def test_the_agent_command_line_does_not_take_them(self) -> None: + agent_line = next(line for line in ENTRYPOINT.read_text().splitlines() if "k3s agent" in line) + self.assertNotIn("etcd", agent_line) + + +if __name__ == "__main__": + unittest.main()