diff --git a/README.md b/README.md index 9937b52..ad6c1f3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ There are two ways to deploy. A fleet's hosts are managed from a control machine with Ansible, from the fleet's own inventory (see [Ansible](#ansible) below). `bootstrap.sh`, described here, brings up a single machine from a checkout of this repository without a control node or inventory. It runs the same Ansible role against the machine itself, so both paths deploy identically. -Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself. Elsewhere, install `helm` and `kubectl` from the host's package manager first. +Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself, and the cluster role starts a Colima host's profile at every login and fails unless the runtime is up (Docker Desktop needs its own start-at-login setting; see Container runtime at login in `roles/github_runner_cluster/README.md`). Elsewhere, install `helm` and `kubectl` from the host's package manager first. 1. **Create a GitHub App** on the organisation with **Self-hosted runners: read & write** permission, and install it on the organisation. `playbooks/github_app_setup.yml` does this through GitHub's manifest flow (see `roles/github_runner_arc/README.md`). 2. **Configure the environment.** Copy `.env.bootstrap.example` to `.env.bootstrap` and fill in the following (the file's own comments cover the rest, such as joining an existing cluster or running as an agent): diff --git a/plugins/filter/container_runtime.py b/plugins/filter/container_runtime.py new file mode 100644 index 0000000..79e6ddb --- /dev/null +++ b/plugins/filter/container_runtime.py @@ -0,0 +1,74 @@ +"""Filter the github_runner_cluster role uses to tell which container runtime a macOS host's Docker context belongs to, and so whether the role can start that runtime itself.""" + +from __future__ import annotations + +import re +from typing import Any + +from ansible.errors import AnsibleFilterError + +RUNTIME_CHOICES = ("auto", "colima", "docker_desktop", "other") +# Colima names the Docker context of its default profile "colima" and any other profile's "colima-". +COLIMA_CONTEXT = "colima" +COLIMA_DEFAULT_PROFILE = "default" +# Docker Desktop's own context for its Linux engine. +DOCKER_DESKTOP_CONTEXTS = ("desktop-linux",) +# Lima's rule for an instance name, which is what a Colima profile becomes. It also keeps the value free of every character that would need escaping in a launchd property list or on a command line. +PROFILE_PATTERN = re.compile(r"^[A-Za-z0-9]+([._-][A-Za-z0-9]+)*$") + + +def _text(name: str, value: Any) -> str: + if value is None: + return "" + if not isinstance(value, str): + raise AnsibleFilterError(f"{name} must be a string, not {type(value).__name__}: {value!r}.") + return value.strip() + + +def _colima_profile_of(context: str) -> str | None: + """Return the Colima profile a Docker context belongs to, or None when it is not a Colima context.""" + if context == COLIMA_CONTEXT: + return COLIMA_DEFAULT_PROFILE + prefix = COLIMA_CONTEXT + "-" + if context.startswith(prefix) and len(context) > len(prefix): + return context[len(prefix) :] + return None + + +def cluster_container_runtime(context: Any, requested: Any = "auto", colima_profile: Any = "") -> dict[str, str]: + """Work out which container runtime a macOS host's Docker daemon belongs to. + + The role calls it only for a macOS host, the only kind it starts a runtime on. ``auto`` takes the runtime from the Docker context the role drives: a Colima context (``colima`` or ``colima-``) is Colima and ``desktop-linux`` is Docker Desktop, and anything else (including no context because ``DOCKER_HOST`` is set) is ``other``. + + :param context: the Docker context the role drives, or an empty value for none. :param requested: ``auto``, ``colima``, ``docker_desktop`` or ``other``. :param colima_profile: the Colima profile to start, or an empty value to take it from the context (``default`` when the context names none). + :returns: ``{"kind": "colima" | "docker_desktop" | "other", "colima_profile": }``. + :raises AnsibleFilterError: if a value is not a string, ``requested`` is not one of the choices, the profile is not a valid Colima profile name, or it names another profile than the Colima context the role drives. + """ + docker_context = _text("The Docker context", context) + choice = _text("The container runtime", requested) or "auto" + profile = _text("The Colima profile", colima_profile) + if choice not in RUNTIME_CHOICES: + raise AnsibleFilterError(f"The container runtime must be one of {', '.join(RUNTIME_CHOICES)}, not {choice!r}.") + if profile and not PROFILE_PATTERN.match(profile): + raise AnsibleFilterError(f"The Colima profile {profile!r} is not a valid profile name: use letters and digits, separated by single dots, hyphens or underscores.") + context_profile = _colima_profile_of(docker_context) + if choice == "auto": + if context_profile is not None: + choice = "colima" + elif docker_context in DOCKER_DESKTOP_CONTEXTS: + choice = "docker_desktop" + else: + choice = "other" + if choice != "colima": + return {"kind": choice, "colima_profile": ""} + if profile and context_profile is not None and profile != context_profile: + raise AnsibleFilterError(f"The Colima profile {profile!r} is not the one the Docker context {docker_context!r} belongs to ({context_profile!r}); the role would start one VM and drive another.") + return {"kind": "colima", "colima_profile": profile or context_profile or COLIMA_DEFAULT_PROFILE} + + +class FilterModule: + """Registers the container runtime filter with Ansible.""" + + def filters(self) -> dict[str, Any]: + """Return the filters this plugin provides.""" + return {"cluster_container_runtime": cluster_container_runtime} diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index f5ac7f4..89cac7e 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -56,6 +56,30 @@ By default the role drives whichever Docker daemon the host's current Docker con Set `github_runner_cluster_docker_context` in the host's variables to pin the context by name (`colima` in that example) on any host with more than one engine. Every `community.docker` module call in the role then passes it as `cli_context`, and every `docker` command the role runs (Compose, `exec`, `inspect`, `volume`, image pulls, and the commands in `playbooks/recover_in_cluster_mesh.yml` and `playbooks/promote_headscale_standby.yml`) runs with `DOCKER_CONTEXT` set to it, so the modules and the CLI always reach the same daemon. The role checks the pin before it touches the host and fails if the context does not exist there, or if `DOCKER_HOST` is also set, since the two would name the daemon in conflicting ways. It never creates a context and never changes the host's current context; `docker context show` on the host reads the same before and after a run. The Headscale tasks that act on another host (the `headscale_hosted` server and standby, and the promotion playbook) use that host's own value. Run `docker context show` on each host to see what it currently points at before choosing a value. +## Container runtime at login (macOS) + +On a Mac the container runtime is a Linux VM (Colima, or Docker Desktop) that stays stopped after a reboot until something starts it. The node's k3s container has `restart: unless-stopped`, so it comes back by itself once the runtime does, but a host whose runtime never starts never rejoins the cluster. With three servers, two such hosts leave the cluster without quorum and no runner pods anywhere. Only macOS hosts are affected by any of this: the role does nothing about the runtime on a Linux host, where Docker is a system service. + +Which runtime a Mac runs is taken from the Docker context the role drives (see [Docker contexts](#docker-contexts)): `colima`, or `colima-` for another profile, is Colima, and `desktop-linux` is Docker Desktop. `github_runner_cluster_runtime` says so explicitly for a host whose context does not tell, for example one that uses `DOCKER_HOST`, and `github_runner_cluster_colima_profile` names the profile to start when the context does not. + +**Colima.** With `github_runner_cluster_runtime_autostart` on (the default, which acts on Colima hosts only), the role: + +- starts the profile if it is stopped, with `colima start --profile --activate=false`, so the profile does not make its context the host's current one; +- installs a launchd agent, `com.exadev.github-runner.colima.`, as `~/Library/LaunchAgents/