clouduler

An opinionated, turnkey alternative to Docker Swarm, built for clusters where the network is secure but unreliable, and where nodes are very different from each other.

It was designed around one real setup:

• a secure but unstable network;
• a cluster that depends on the network, and a network that depends on the cluster;
• a scattered topology with several public IPs of different capabilities;
• heterogeneous storage, RAM and CPU.

The goal was to beat the stability of Swarm + Traefik + GlusterFS/Ceph/SeaweedFS on that infrastructure, and it has been met: no crash or downtime caused by clouduler since. Every crash found during development got a regression test where one could apply, and an end-to-end test harness based on QEMU replays failures on clusters of different sizes to check that they always recover.

It ships as one binary (clouduler, about 65 MB on arm64) with the web UI embedded. Few external dependencies: Docker, restic for backups, and the nfsv4 kernel module to access remote volumes without a local copy.

Features

• Containers. Swarm-like orchestration with better docker compose compatibility, plus Swarm-only features missing from compose (configs). Most behaviour is configurable per stack or per container.
• Replicated drives. A drive is a folder of volumes, replicated in full across the nodes that implement it (a node implements a drive by designating a folder, usually on its own storage). Data stays local whenever possible.
• Reverse proxy. Uses Cloudflare DNS for Let's Encrypt DNS-01 certificates and routes each request to the right node, since nodes can have different public and internal addresses. It can also expose services that clouduler does not manage, such as a box admin page.
• Networks. Access to web services is governed by networks: one public network, and any number of private ones, each tied to a subnet and its own domain (for example liteapp.fr and home.liteapp.fr). Someone outside the subnet cannot reach home.liteapp.fr. The same rule applies to containers: one that is not on the internal network cannot reach its services.
• Backups. Per-drive restic backups to one or more cluster nodes, SSH (scp) or S3, with the usual retention policies. Each backup server has its own encryption key, usable with plain restic. Restore a volume from the web UI.
• GitOps. Configuration is pushed to and pulled from git. Configs are part of it, so they can be restored without any data, and editing a config inside a container can commit automatically.
• Updates. Quick updates from Docker Hub or private registries.
• Management. Web UI (OIDC or API key) for almost everything, and a CLI that can docker exec from any node and reach advanced features like freezing a volume so replication does not interfere while you work on it.

Security

The network the nodes share is assumed to be secure (VPN or internal VLAN), so there is no extra encryption layer. Container traffic is plain text (VXLAN), and node-to-node administration traffic is plain text too, but authenticated with signatures.

• Secrets are stored in raft, encrypted with XChaCha20-Poly1305 under a cluster key. The key is only shared at pairing, encrypted to the joining node's public key, and each node keeps it in a mode 600 file.
• Secrets in git are encrypted with age to a public key provided at setup, so clouduler only ever holds the public key.
• Per-stack access control. Each stack declares which networks it can reach (private, public, internet) and can restrict each one, for example only auth.liteapp.fr for OIDC and *.github.com for updates. Clouduler is the containers' DNS resolver: when a container asks for api.github.com, the firewall rule is added before the answer is returned.
• Internal HTTP(S) requests go through the reverse proxy, which enforces access. Other protocols (mail, for instance) must be allowed explicitly and are added to the firewall at start.

The aim is to treat stacks as small, well-guarded islands that are hard to move out of.

Security model: the internal network is expected to be secure. Most traffic between nodes (raft, replication, overlay) is not encrypted, because clouduler is designed to run over an already encrypted VPN such as Tailscale or Headscale. Do not expose the cluster subnet to untrusted networks.

Quick start (from a release)

Requirements on every node: Linux with systemd, Docker, root access. Optional: restic (volume backups). All nodes must reach each other over one trusted internal network (LAN, WireGuard, Tailscale or Headscale). Clouduler does not encrypt that traffic. Every node's internal address must be inside the subnet given to --subnet when the cluster is created; a node outside it cannot join.

Release binaries are named clouduler-x86_64-unknown-linux-gnu and clouduler-aarch64-unknown-linux-gnu and are published by the release workflow on every v* tag.

1. Install the binary.

VERSION=vX.Y.Z   # a release tag
ARCH=$(uname -m)-unknown-linux-gnu
# download clouduler-$ARCH from the release page of the repository, then:
sudo install -m 0755 clouduler-$ARCH /usr/local/bin/clouduler
clouduler --help

2. Create the first node. Interactive:

sudo clouduler init

Or scripted:

sudo clouduler init --yes \
  --cluster-name home \
  --subnet 10.10.0.0/24 \
  --internal-domain int.example.com \
  --web-address cloud \
  --drive fast=/srv/clouduler/fast

This writes /etc/clouduler/cluster.yaml, creates the drive directories, installs and starts clouduler.service.

3. Save what is logged on the very first start (shown once). The API key is only needed to sign in to the web UI before OIDC is set up, and as a fallback when OIDC fails:

journalctl -u clouduler | grep -i -E "api key|fingerprint"

• the generated admin API key (clk_...)
• the cluster key fingerprint

4. Back up /etc/clouduler/secrets.key right now (see Keys below).

5. Open https://cloud.int.example.com:8443 and sign in with the API key. The certificate is self-signed until Cloudflare and ACME are set up in the next step.

6. Set up Cloudflare (recommended), in the web UI, before adding other nodes. Put your domain's DNS on Cloudflare and create an API token with Zone:DNS:Edit on that zone. In the UI, set the Cloudflare zones and the ACME settings (contact email; start on the staging directory and switch to production once it works), and enter the API token in the Cloudflare settings. Clouduler then gets real certificates through ACME DNS-01, which needs no inbound access, so the UI and API serve a trusted certificate and you never need --tls-fingerprint when joining nodes. Config changes apply live, no restart needed. Wait for the certificate before continuing. Without Cloudflare the cluster keeps its self-signed certificate and joins should be pinned by fingerprint.

   The file equivalent, in /etc/clouduler/cluster.yaml (see examples/cluster.yaml):

cloudflare:
  zones:
    - example.com
proxy:
  acme:
    enabled: true
    contact_email: ops@example.com
    # start on staging, remove directory_url for production once it works
    directory_url: https://acme-staging-v02.api.letsencrypt.org/directory

7. Add more nodes. Open the Nodes page in the web UI and add a node: it generates the exact join command to run on the new machine, so you don't need to write it by hand. The new node's address must be in the cluster subnet. Run the generated command on the machine with the binary installed.

Use at least 3 managers for a fault-tolerant cluster (--role manager). The cluster key is handed to the new node automatically, wrapped for its own keypair.

8. Check health: clouduler check (parse config), clouduler status (live problems, ends with all clear).

9. Deploy something: clouduler stack deploy with a file like examples/stacks/simple.yaml, or use the UI.

Keys and credentials to set up

What	Where	Needed for	Notes
Cluster key	/etc/clouduler/secrets.key (secrets.key_file), base64, mode 0600	Sealing secrets (XChaCha20-Poly1305) and authenticating node-to-node traffic	Generated on the first node, auto-distributed to joiners. If every copy is lost, all stored secrets are lost. Back it up, or set up git backup.
Admin API key	Logged once on first launch; more from the UI	Signing in to the web UI before OIDC is set up, or when OIDC fails; CLI, headless MCP, CI. Not needed for joining nodes (the Nodes page generates the join command)	Sent as Authorization: Bearer clk_.... Only a hash is stored.
TLS fingerprint	Printed by init --join	Verifying the node you join while the cluster still serves a self-signed certificate	Not needed once Cloudflare DNS-01 is set up and a real certificate is issued.
OIDC (recommended for people)	auth.oidc in config (issuer, client_id, admin_claim, admin_value); client secret entered in the UI	Single sign-on; users with admin_value in admin_claim become admin, others viewers	Keep auth.api_keys: true until OIDC is proven to work.
Cloudflare API token	Entered in the UI (stored as a cluster secret); zones in the UI or cloudflare.zones	Public DNS records and ACME DNS-01 certificates	Set up in the UI after init, before adding other nodes. Needed for proxy.acme.enabled: true.
ACME contact email	proxy.acme.contact_email	Let's Encrypt expiry notices	Starts on the staging directory; switch directory_url when ready.
Git backup: age key + deploy key	age public key in backup.age_recipient; deploy key generated by the cluster	Off-cluster backup of config, stacks, configs, secrets	Create the age key with age-keygen outside the cluster. Add the displayed public deploy key to the repo with write access.
Registry credentials	UI, Registry accounts	Pulling private images	Scoped per registry.
MCP	mcp.enabled: true	AI clients	Off by default; OAuth consent, read-only or admin.

Minimal config is in examples/cluster-minimal.yaml; every option is in examples/cluster.yaml.

Running over Tailscale or Headscale

Clouduler works over a Tailscale or Headscale tailnet; this is what the author uses. Use the tailnet addresses as the cluster's internal network: pass the tailnet range (for example 100.64.0.0/10, or a narrower range) as --subnet, and make sure every node's tailnet address is inside it. The interface must already be up before clouduler starts (see docs/networking.md).

The tailnet is what provides encryption for node-to-node traffic.

With Headscale, the order matters, because Headscale itself can run as a clouduler stack:

1. Deploy the Headscale container first, on a cluster that is reachable some other way (LAN or a temporary network).
2. Connect every other node to the tailnet.
3. Only then add those nodes to the cluster, using their tailnet addresses.

Do not move the cluster onto a tailnet that depends on a Headscale instance the cluster itself has not yet started.

Recovering when the Headscale container is lost

If Headscale goes down, the nodes lose the tailnet, and with it the quorum the cluster needs to restart Headscale. That is a deadlock, and clouduler emergency exists for exactly this case.

1. On one manager that still has its local data, stop the daemon: sudo systemctl stop clouduler.
2. Run sudo clouduler emergency headscale --yes (the argument is the stack to bring back up; --node defaults to the hostname).
3. Start the daemon: sudo systemctl start clouduler.

The node becomes the sole voter and starts the Headscale stack from its own local data. Other stacks are held off, not stopped. Once Headscale is back and the other managers reconnect, they are promoted back to voters and emergency mode ends on its own. If the stack's content itself is gone, run clouduler restore afterwards from the git backup (see docs/git-backup.md).

How it works

Architecture

Managers vote in raft and hold the replicated database. Workers keep a read-only replica and only run containers. Every node runs a scheduler that reconciles local Docker against the replicated state, plus the reverse proxy and API.

Source: docs/.diagrams/architecture.d2. Details: database, networking.

Deploying a stack

A write can go to any node and is forwarded to the raft leader. Once a majority of managers have it, it is applied identically everywhere, and each node's scheduler starts or stops what belongs on it. Secrets are sealed before they reach the log, so the machine running stack deploy needs the cluster key.

See stacks, secrets, drives.

Joining a node

The joiner is not in raft yet, so it goes through the HTTP API with an admin key, confirms the config edit (otherwise it rolls back), and receives the cluster key wrapped for its own public key.

See setup.

Failover and fencing

Two timers work against each other: an isolated node stops its own containers after fence_timeout, and only after node_timeout does the leader reschedule them elsewhere. probe_interval < fence_timeout < node_timeout is enforced so a volume never has two writers.

See failover. Disaster recovery: clouduler emergency restores quorum, clouduler restore restores content from git backup.

Documentation

• setup: init, joining, systemd, config replication
• stacks: stack file format and x-clouduler extensions
• secrets: configs, secrets, cluster key
• proxy: reverse proxy, DNS, certificates
• networking: overlay, egress networks
• drives, served drives, volume backups, git backup
• failover, live config, alerts
• api, mcp, ui, testing

Building from source

cd ui && npm ci && npm run build && cd ..
cargo build --release -p clouduler

Regenerating diagrams

cd docs/.diagrams && for f in *.d2; do d2 --layout=elk $f ${f%.d2}.svg; done