From adf0cdab052950ab87666b6170403aee8110b4ec Mon Sep 17 00:00:00 2001 From: Lorenzo Corallo Date: Fri, 9 Oct 2026 21:16:57 +0200 Subject: [PATCH 1/4] docs(infrastructure): document K3s and Flux migration - Replace AKS-era guidance with the Terraform, Ansible, K3s, and Flux setup - Add node provisioning, GitOps, and encrypted backup documentation - Update access, storage, monitoring, CI/CD, and app deployment guides --- docs/infrastructure/01-Introduction.md | 42 +- .../cloud-infrastructure/01-terraform.md | 118 ++-- .../cloud-infrastructure/02-k3s-node.md | 139 +++++ .../cloud-infrastructure/03-flux.md | 148 +++++ .../cloud-infrastructure/index.md | 85 ++- .../k8s-basics/day-to-day/monitoring.md | 107 ++-- .../k8s-basics/day-to-day/ssh.md | 12 +- .../02-getting-started/k8s-basics/intro.md | 13 +- .../02-getting-started/pipelines/cd.md | 16 +- .../02-getting-started/pipelines/ci.md | 10 +- .../02-getting-started/setup.md | 101 ++- .../03-Guides/add-a-new-secret.md | 585 +++++++----------- docs/infrastructure/03-Guides/add-storage.md | 136 ++-- docs/infrastructure/03-Guides/backups.md | 154 +++++ .../03-Guides/deploy-a-new-app.md | 441 ++++++++----- 15 files changed, 1396 insertions(+), 711 deletions(-) create mode 100644 docs/infrastructure/02-getting-started/cloud-infrastructure/02-k3s-node.md create mode 100644 docs/infrastructure/02-getting-started/cloud-infrastructure/03-flux.md create mode 100644 docs/infrastructure/03-Guides/backups.md diff --git a/docs/infrastructure/01-Introduction.md b/docs/infrastructure/01-Introduction.md index b86d307..1a4fb43 100644 --- a/docs/infrastructure/01-Introduction.md +++ b/docs/infrastructure/01-Introduction.md @@ -11,24 +11,50 @@ Let's discover **PoliNetwork's Infrastructure in less than 5 minutes**. ### Where Our infrastructure is deployed on Azure West Europe (Amsterdam). -The description is found [here](https://github.com/polinetworkorg/terraform), described in Terraform (IaC). +The Azure resources are described in Terraform (IaC) in the +[terraform](https://github.com/polinetworkorg/terraform) repository. ### What -The core is a AKS Cluster (Kubernetes), in combination with various other services (KeyVault, Disks, Azure Active Directory, ...) +Production runs on **K3s**, a lightweight Kubernetes distribution, installed on +a single Azure ARM64 VM called `k3s01`. Around it there are a few Azure services +(Key Vault, Managed Disks, Blob Storage, Entra ID) and Cloudflare, which +publishes our websites through a tunnel and gives us SSH access to the VM. + +Three layers, three tools: + +| Layer | Tool | Repository | +|-------|------|------------| +| Azure resources (VM, disks, network, Key Vaults, identities) | Terraform | [terraform](https://github.com/polinetworkorg/terraform) | +| The VM itself (disks, firewall, K3s, backups) | Ansible | [polinetwork-cd/ansible](https://github.com/polinetworkorg/polinetwork-cd/tree/main/ansible) | +| Everything running inside Kubernetes | Flux (GitOps) | [polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) | ### Why -Why not a simple VM? -AKS gives us the flexibility of a 10s pod deployment, reliability (with ArgoCD ensuring all pods are up and running all the time) and allows us to be more precise and clean in our infrastructure, with better monitoring and allocated resources for each application depending on the importance. +Why Kubernetes on a single VM instead of plain Docker? +Kubernetes gives us declarative deployments, rolling updates, health checks and +per-application resource limits. Flux keeps the cluster in sync with Git, so the +`polinetwork-cd` repository is always the source of truth for what is running. + +Why K3s instead of a managed cluster? +Until October 2026 we ran on AKS (Azure Kubernetes Service). K3s gives us the +same Kubernetes API on one VM that we fully control and size ourselves, and it +drops our dependency on AKS-specific add-ons (Azure CSI drivers, Longhorn, ArgoCD). + +:::note Legacy AKS +The AKS cluster `aks-polinetwork` is still defined in the `environments/legacy` +Terraform root, but no production workload runs on it anymore. +If you find instructions mentioning `az aks get-credentials`, ArgoCD, +`SecretProviderClass` or Longhorn (e.g. in old READMEs), they refer to the AKS setup. +::: ## About this documentation :::important -Most of the docs explain procedures with the Azure CLI and the Kubernetes CLI. -Make sure to [install and configure them](./getting-started/setup#azure-cli--kubectl) to follow along. -Alternatively, you can see the [official Azure documentation](https://learn.microsoft.com/en-us/azure/?product=popular) for how to use the Azure portal. +Most of the docs explain procedures with `kubectl`, which you run on the K3s node +over SSH, and with the Azure CLI, which you use for Key Vault secrets. +Make sure to [set up your access](./getting-started/setup) to follow along. ::: The best place to start is going through the [Getting Started](./getting-started/setup) section to get a high level overview of how things are organized. -If you're looking for a guide on how to achieve a specific outcome, you can look in the [Guides](./Guides/add-a-new-secret). +If you're looking for a guide on how to achieve a specific outcome, you can look in the [Guides](./Guides/deploy-a-new-app). diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md index 1fcf7d3..d4a5799 100644 --- a/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/01-terraform.md @@ -18,16 +18,18 @@ local machine. If you need help setting up Terraform, please refer to the You'll also need to be logged in with the Azure CLI and have the necessary permissions to manage the resources in our Azure subscription (more on this -[here](../setup#azure-cli--kubectl)). +[here](../setup#azure-cli)). -Once you have logged in to the Azure CLI, you need to set the necessary -environment variables by running this command at the root of the repository: +Once you have logged in to the Azure CLI, select the subscription and export it +for the provider: ```sh -source ./access_key.sh +az account set --subscription +export ARM_SUBSCRIPTION_ID= +export TF_VAR_subscription_id="$ARM_SUBSCRIPTION_ID" ``` -More about this in the [repository README](https://github.com/PoliNetworkOrg/terraform/?tab=readme-ov-file#notes). +More about this in the [repository README](https://github.com/PoliNetworkOrg/terraform/#local-checks). ::: ### Why We Use Terraform @@ -46,7 +48,7 @@ To run terraform commands you need to have the right permissions and credentials If you are not sure about your permissions, please contact the IT department (but if you have to ask you probably shouldn't run `apply` yourself anyway). -If you haven't yet set up the Azure CLI please refer to the [setup guide](../setup#azure-cli--kubectl). +If you haven't yet set up the Azure CLI please refer to the [setup guide](../setup#azure-cli). ::: Terraform operates through a simple yet powerful workflow: @@ -70,7 +72,7 @@ Always review the execution plan (`terraform plan`) before applying changes. infrastructure. These files are the blueprint for what you want the infrastructure to look like. - **Terraform State:** - The state file (which we named `state.tfstate`) records the current state + The state file (we have two of them, see [State Management](#state-management)) records the current state of the infrastructure as managed by Terraform. This file is critical because it maps your configuration to the real-world resources, tracks dependencies, and allows Terraform to detect what needs to change during updates. Our state @@ -101,18 +103,29 @@ and our setup is tailored to our current needs. ### Repository Overview The **polinetworkorg/terraform** repository is our centralized solution for -provisioning and managing cloud infrastructure on Azure. It follows a modular -design that breaks down the overall architecture into focused components. To give -you a rough idea of what is in there, here are some of the primary modules: - -- **aks:** Defines all resources related to our Azure Kubernetes Service (AKS) instance. -- **mariadb:** Manages the main MariaDB database. -- **monitoring:** Sets up our monitoring stack using Grafana and Prometheus. -- **argocd:** Configures our ArgoCD installation for GitOps-driven deployments. -- **storage:** Handles the Azure Storage Account where the Terraform state is stored. - -There are also additional modules in the repository that handle specific -applications or services, but these are not detailed in this general overview. +provisioning and managing cloud infrastructure on Azure. Its default branch is +**`stable`**. + +The code is split into two independent **root modules** (the folders you run +`terraform` in). Each root owns its resources and has its own state file; a root +can only read the other's resources through data sources, never declare them. + +- **`environments/k3s`:** the production K3s platform. It defines: + - the `k3s01` VM (Debian 13 ARM64) and its two data disks, `disk-k3s-fast` + (Premium SSD v2, 64 GB) and `disk-k3s-standard` (Standard SSD, 128 GB); + - the network: `vnet-k3s` (`10.43.0.0/16`), a NAT Gateway for outbound traffic, + and an NSG that denies all inbound traffic; + - the Key Vaults `kv-pn-apps` (application secrets) and `kv-pn-infra` + (platform secrets); + - the managed identities, including the one External Secrets uses through + workload identity federation, and the public OIDC issuer the cluster's + service-account tokens are verified against. +- **`environments/legacy`:** everything from before the migration: the old AKS + cluster and its Kubernetes/Helm resources (ArgoCD, Longhorn, Kubernetes + Dashboard, MariaDB, ...), plus shared resources such as the resource group + `rg-polinetwork`, the backup storage account `polinetworkbackups` and the budgets. + +What runs *inside* the VM is not managed here: see [K3s Node](./02-k3s-node.md) and [Flux](./03-flux.md). :::info While the choices made in this implementation reflect our specific requirements @@ -132,25 +145,32 @@ While the exact folder structure may vary, a typical layout looks like this: ```bash 📁 polinetworkorg/terraform/ - ├─ 📁 modules/ # All modules are kept in a dedicated directory - │ ├─ 📁 aks/ # Module for Azure Kubernetes Service resources - │ ├─ 📁 mariadb/ # Module for the main MariaDB database - │ ├─ 📁 monitoring/ # Module for monitoring (Grafana + Prometheus) - │ ├─ 📁 argocd/ # Module for ArgoCD installation - │ ├─ 📁 storage/ # Module for Azure Storage Account where the state.tfstate is kept + ├─ 📁 environments/ # Root modules: run terraform here, with -chdir + │ ├─ 📁 k3s/ # K3s node, network, Key Vaults, identities (k3s.tfstate) + │ └─ 📁 legacy/ # Old AKS platform and shared resources (state.tfstate) + ├─ 📁 modules/ # Modules used by the legacy root + │ ├─ 📁 aks/ # Azure Kubernetes Service + │ ├─ 📁 argocd/ # ArgoCD installation + │ ├─ 📁 shared/ # Backup storage account, budgets + │ ├─ 📁 storage/ # Azure Storage Account where the Terraform state is kept │ └─ [other modules] # Additional modules for specific services - ├── main.tf # Main Terraform configuration file - ├── variables.tf # Variables definition file - ├── outputs.tf # Outputs definition file ├── README.md # General repository documentation - └── [other files] + ├── STATE_MIGRATION.md # How the state was split between the two roots + └── [other files] ``` ### State Management We use an **Azure Storage Account** for managing the Terraform state. The state -file (`state.tfstate`) is stored within a dedicated container named `terraform-state`, -as defined in our **storage** module. The integrated leasing mechanism provided +files are stored within a dedicated container named `terraform-state`, +as defined in our **storage** module: + +| Root | State file | +|------|------------| +| `environments/legacy` | `state.tfstate` | +| `environments/k3s` | `k3s.tfstate` | + + The integrated leasing mechanism provided by the Azure Storage Account is used for state locking, ensuring that concurrent Terraform operations do not conflict with each other. @@ -190,14 +210,18 @@ consistency and control. 1. **Edit the Terraform Code:** Modify the relevant configuration files to define your desired changes in the - infrastructure. + infrastructure. Most changes belong to `environments/k3s`. 2. **Review Your Changes Locally:** - Run `terraform plan` to generate an execution plan. This command will show - you what changes will be made to align the actual infrastructure with your - code. + Run `terraform -chdir=environments/ plan` to generate an execution plan. + This command will show you what changes will be made to align the actual + infrastructure with your code. + +3. **Open a Pull Request against `stable`:** + The **Terraform workflow** runs format, init, validate and plan for both roots + and posts one comment per root with the plan. Pull requests never apply. -3. **Cost Estimation with Infracost:** +4. **Cost Estimation with Infracost:** When you open a pull request, our **Infracost workflow** is triggered automatically. This workflow calculates the estimated cost impact of your changes and posts a report in the pull request comments, so you can review @@ -207,24 +231,30 @@ consistency and control. [official documentation](https://www.infracost.io/docs/). ::: -4. **Apply the Changes:** - After your changes have been reviewed and approved, merge the pull request - and run `terraform apply` to implement the changes. This command updates the - Azure resources to match the configuration defined in your code. +5. **Apply the Changes:** + After your changes have been reviewed and approved, merge the pull request. + The push to `stable` runs the plans again, then waits for a maintainer to + approve the GitHub **`production`** environment. Once approved, it applies + `k3s` first and `legacy` second. You don't run `terraform apply` yourself. -5. **Drift Detection:** +6. **Drift Detection:** A nightly **Drift workflow** runs to detect any discrepancies between the current Terraform state and the actual state in Azure. If this workflow detects a drift from what the `terraform plan` output would predict, it automatically opens an issue for further investigation. + At the moment it only checks the `legacy` root. ### Basic Terraform CLI Commands +Always pick a root with `-chdir`, e.g. `terraform -chdir=environments/k3s plan`. +Running Terraform at the repository root does nothing useful. + - **`terraform init`** Initializes the working directory containing the Terraform configuration files. :::tip This has to be run as soon as you clone the repository or when you pull changes - that include new modules or providers. + that include new modules or providers. Locally, add + `-backend-config=use_oidc=false` (OIDC is only for GitHub Actions). ::: - **`terraform plan`** @@ -232,13 +262,15 @@ consistency and control. - **`terraform apply`** Applies the changes defined in your Terraform configuration to the cloud environment. + In practice the Terraform workflow does this for you after the `production` approval. - **`terraform destroy`** Destroys the resources managed by Terraform, useful for tearing down test environments or decommissioning infrastructure. :::danger If you ever happen to be as much as tempted of using `terraform destroy`, you - better be **SURE AS SHIT** about what you are doing. Make sure to review the + better be **SURE AS SHIT** about what you are doing. Never run it without + `-chdir`, and remember the K3s data disks have `prevent_destroy` for a reason. Make sure to review the implications thoroughly before proceeding. > *One does not simply destroy PoliNetwork with one command.* ::: diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/02-k3s-node.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/02-k3s-node.md new file mode 100644 index 0000000..5a72668 --- /dev/null +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/02-k3s-node.md @@ -0,0 +1,139 @@ +--- +title: "K3s Node" +description: "The k3s01 VM, how Ansible configures it, and how to run the playbooks." +--- + +## Overview + +Production runs on a single VM, **`k3s01`**, created by Terraform +(`environments/k3s`) and configured by the Ansible playbooks in +[`polinetwork-cd/ansible`](https://github.com/polinetworkorg/polinetwork-cd/tree/main/ansible). + +| | | +|---|---| +| Size | `Standard_E2ps_v6` (ARM64), West Europe, zone 1 | +| OS | Debian 13 ARM64 | +| Private IP | `10.43.1.4` (no public IP) | +| Kubernetes | K3s, single server node, pinned version (see `ansible/vars/main.yml`) | +| Admin user | `pnadmin` | + +:::important ARM64 +The node is ARM64. Every image we deploy must have a `linux/arm64` variant, +otherwise the pod fails with `exec format error`. Our GitHub workflows publish +multi-arch images (`amd64` + `arm64`) for this reason. +::: + +## Disks + +| Mount | Azure disk | Type | Used for | +|-------|-----------|------|----------| +| `/` | `disk-k3s01-os` | Standard SSD, 64 GB | Operating system | +| `/srv/fast` | `disk-k3s-fast` (LUN 0) | Premium SSD v2, 64 GB | Volumes of the `fast` storage class (databases) | +| `/srv/standard` | `disk-k3s-standard` (LUN 1) | Standard SSD, 128 GB | K3s data (`/srv/standard/k3s`), volumes of the `standard` storage class, local backup staging | + +Ansible finds the data disks by Azure LUN, refuses to touch a disk that is +smaller than expected or holds an unexpected filesystem, formats them only when +blank and mounts them by UUID. Both disks have `prevent_destroy` in Terraform. + +See [Add Storage](../../Guides/add-storage) for how volumes are carved out of these disks. + +## What Ansible sets up + +The provisioning playbook (`ansible/playbooks/provision.yml`) applies these roles, in order: + +| Role | What it does | +|------|--------------| +| `base` | Base packages, UTC timezone, `pnadmin` user and the `k3s-admin` group, kernel modules and sysctls for Kubernetes | +| `security` | SSH hardening (keys and Cloudflare Access certificates only), Debian security updates via `unattended-upgrades` (no automatic reboot), `nftables` host firewall | +| `storage` | Formats and mounts the data disks, creates `/srv/*/volumes` | +| `k3s` | Installs the pinned K3s binary (checked by SHA-256) and its configuration, waits for the node to be `Ready` | +| `backup` | Installs the encrypted backup scripts and their systemd timers (see [Backups](../../Guides/backups)) | +| `flux` | Installs the pinned Flux Operator and the `FluxInstance` that syncs `clusters/k3s` from the `main` branch | + +After that, Flux takes over everything inside the cluster (see [Flux](./03-flux.md)). + +### Notable K3s settings + +- **Pod and Service CIDRs** are `10.52.0.0/16` and `10.53.0.0/16`. The K3s + default `10.43.0.0/16` would collide with the Azure VNet. +- **ServiceLB** and the bundled **local-storage** are disabled. Storage comes from + the local-path provisioner managed by Flux. +- The bundled **Traefik** stays as the ingress controller, but Flux turns its + Service into `ClusterIP`: only `cloudflared` reaches it. +- **Secrets encryption** at rest is enabled. +- The API server signs service-account tokens for a public OIDC issuer + (`pnk3soidc` storage account). Azure trusts that issuer, which is how External + Secrets reads Key Vault without any stored credential. + +### Security guardrails + +- The VM has no public IP and the NSG denies all inbound traffic. The host + firewall drops everything except SSH, the node itself and cluster traffic. +- Pods cannot reach the Azure Instance Metadata Service (`169.254.169.254`), so + they can't borrow the VM's managed identities. +- An admission policy rejects pods using `hostNetwork`, `hostPID` or `hostIPC` + outside `kube-system`, and another one forbids pods from running as the + `keyvault-reader` service account. + +## Running the playbooks + +:::warning +You rarely need this: day-to-day changes go through Flux. Run Ansible when you +change something under `ansible/` (e.g. a K3s upgrade or a new backup target). +Only the Direttivo has access to the node. +::: + +The playbooks run from your machine against `10.43.1.4` over SSH, so connect +[WARP](../setup#cloudflare-warp--ssh) first. + +```sh +cd ansible +ansible-galaxy collection install -r requirements.yml +./scripts/provision-and-verify.sh +``` + +The script is the only supported way to apply changes. It runs the playbook in +check mode, then applies it twice and fails if the second run still changes +something (the playbook must be idempotent). Finally it runs +`playbooks/verify.yml`, which checks the disks, the K3s version, the firewall, +the SSH configuration, that Flux is ready, that pods can't reach Azure IMDS, and +it takes and validates a fresh backup. Logs are saved in `ansible/artifacts/` +(ignored by Git). + +To run from inside the VM (for example through Azure Run Command), put a reviewed +checkout on the VM and run: + +```sh +./scripts/provision-and-verify.sh inventories/local/hosts.yml +``` + +:::danger +Never copy a private SSH key or a secret value into the repository or into Azure +Run Command parameters. +::: + +### CI + +The **Ansible** workflow in polinetwork-cd runs on every pull request that +touches `ansible/`, `clusters/k3s/`, `infrastructure/`, `apps/` or `tests/`. It +runs `ansible-lint`, `shellcheck`, a playbook syntax check, `kustomize build` on +every Flux path and the manifest tests in `tests/`. It does **not** touch the VM. + +### Upgrading K3s + +1. Pick a release from [K3s releases](https://github.com/k3s-io/k3s/releases) and + download its `k3s-arm64` binary checksum. +2. Update `k3s_version`, `k3s_release_url` and `k3s_sha256` in `ansible/vars/main.yml`. +3. Open a PR, merge it, then run `./scripts/provision-and-verify.sh`. K3s restarts + with the new binary; workloads come back on their own. + +The Flux Operator is upgraded the same way (`flux_operator_*` and +`flux_distribution_version`). + +:::caution Rebuilding the node +The workload identity trust relies on the cluster's service-account signing key, +whose public part is committed in Terraform +(`environments/k3s/k3s-service-account-jwks.json`). If the node is rebuilt from +scratch without restoring the control-plane backup, the key changes and that file +must be updated, otherwise External Secrets can't read Key Vault. +::: diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/03-flux.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/03-flux.md new file mode 100644 index 0000000..730e35f --- /dev/null +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/03-flux.md @@ -0,0 +1,148 @@ +--- +title: "Flux" +description: "How Flux deploys the polinetwork-cd repository to the K3s cluster." +--- + +## What is Flux? + +[Flux](https://fluxcd.io/) is a GitOps tool: it runs inside the cluster, watches a +Git repository and applies whatever it finds there, over and over. If someone +changes a resource by hand, Flux puts it back the way Git says. It replaced the +ArgoCD instance we had on AKS. + +Ours is installed by Ansible through the +[Flux Operator](https://fluxcd.control-plane.io/operator/), and it syncs the +**`main`** branch of +[polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd), starting +from the `clusters/k3s` folder. + +:::important +Merging to `main` in polinetwork-cd **is** deploying to production. +Never `kubectl apply` or `kubectl edit` a resource that Flux manages: Flux reverts +it at the next reconciliation. +::: + +## Repository layout + +```bash +📁 polinetworkorg/polinetwork-cd/ + ├─ 📁 ansible/ # Host configuration (see K3s Node) + ├─ 📁 clusters/k3s/ # Entry point: one Flux Kustomization per component + │ ├─ infrastructure-*.yaml # Platform components + │ └─ 📁 apps/ # One file per app + kustomization.yaml listing them + ├─ 📁 infrastructure/ # Manifests of the platform components + ├─ 📁 apps/ # Manifests of the apps, one folder per namespace + │ ├─ 📁 backend/ + │ ├─ 📁 web/ + │ └─ ... + └─ 📁 tests/ # Manifest checks run in CI +``` + +Folders at the root of the repository (`bot-prod/`, `bot-rooms/`, `mariadb/`, ...) +are leftovers from the AKS era, all disabled. Flux ignores them. + +### `clusters/k3s` + +Each file here is a Flux +[`Kustomization`](https://fluxcd.io/flux/components/kustomize/kustomizations/) +that points to one folder of `infrastructure/` or `apps/`. It also sets: + +- `dependsOn`: what must be ready first. Apps wait for the secret stores, storage + and Traefik; `backend` also waits for `postgres` and `redis`. +- `prune: true`: a resource removed from Git is deleted from the cluster. +- `wait: true`: the Kustomization is ready only when all its resources are healthy. +- `interval: 30m`: a full reconciliation at least every 30 minutes, even if Git + didn't change. + +`clusters/k3s/apps/kustomization.yaml` lists the apps Flux runs. An app file +that is not listed there does nothing. + +### Platform components (`infrastructure/`) + +| Component | What it does | +|-----------|--------------| +| `security` | Admission policies (no host namespaces, no pods as `keyvault-reader`) | +| `storage` | [local-path provisioner](https://github.com/rancher/local-path-provisioner) and the `fast` / `standard` storage classes | +| `traefik` | Makes the bundled Traefik a `ClusterIP` service | +| `external-secrets` | [External Secrets Operator](https://external-secrets.io/), installed with Helm | +| `secret-stores` | The `kv-pn-apps` and `kv-pn-infra` `ClusterSecretStore`s | +| `cloudflared` | The Cloudflare Tunnel connector (2 replicas) | +| `image-automation` | Detects new `latest` images on GHCR (see below) | +| `node-exporter` | Node metrics for Prometheus | +| `flux-web` | The Flux Web UI at [flux.polinetwork.org](https://flux.polinetwork.org) | + +## Automatic image updates + +Most of our apps are deployed from the `latest` tag of an image on +[GHCR](https://github.com/orgs/PoliNetworkOrg/packages). A tag can point to a +different image over time, so Flux resolves it to a **digest** and deploys that +digest. When the digest changes, the app rolls out the new image. + +```mermaid +flowchart LR + ci[App CI pushes :latest] --> gh[GitHub package event] + gh -- webhook --> rcv[Flux Receiver] + rcv --> prov[ResourceSetInputProvider
resolves latest to a digest] + prov --> rs[ResourceSet
renders the app Kustomization] + rs --> app[Deployment rolls out] +``` + +The pieces, all in polinetwork-cd: + +- `infrastructure/image-automation/image-providers.yaml`: one + `ResourceSetInputProvider` per image, which reads the `latest` tag and its digest. +- `infrastructure/image-automation/receiver.yaml`: a Flux `Receiver` called by + the PoliNetworkOrg organization webhook on `package` events, at + `flux-webhook.polinetwork.org`. The webhook's HMAC signature is its only + authentication. If the webhook doesn't fire, the providers poll every 6 hours anyway. +- `clusters/k3s/apps/.yaml`: for apps with automatic updates this is a + `ResourceSet` instead of a plain Kustomization. It generates the app's + Kustomization with an `images` override that pins the digest. + +Apps that don't need automatic updates (databases, monitoring, ...) pin their +image in the manifest, and you bump them with a PR. + +## Day-to-day + +### Web UI + +[flux.polinetwork.org](https://flux.polinetwork.org) shows every Kustomization, +HelmRelease and ResourceSet, with its status, last applied revision and errors. +You log in through [auth.polinetwork.org](https://auth.polinetwork.org); only +the Direttivo is admitted. From the UI you can **reconcile** (sync now), +**suspend** and **resume** any resource. + +### From the node + +The `flux` CLI isn't installed on the node, but the Flux resources are regular +Kubernetes objects: + +```sh +# Status of all the Kustomizations (apps and infrastructure) +kubectl get kustomizations -n flux-system + +# Latest image resolved for each app +kubectl get resourcesetinputproviders -n flux-system + +# Why is an app not ready? +kubectl describe kustomization -n flux-system + +# Sync now instead of waiting: fetch the latest commit, then apply it +kubectl annotate --overwrite gitrepository flux-system -n flux-system \ + reconcile.fluxcd.io/requestedAt="$(date +%s)" +kubectl annotate --overwrite kustomization -n flux-system \ + reconcile.fluxcd.io/requestedAt="$(date +%s)" +``` + +:::tip Suspending an app +To stop Flux from touching an app while you debug it by hand, **suspend** its +Kustomization from the Web UI. Remember to **resume** it afterwards: a suspended +app also stops receiving image updates. +::: + +## References + +- [Flux Documentation](https://fluxcd.io/flux/) +- [Flux Operator Documentation](https://fluxcd.control-plane.io/operator/) +- [ResourceSets](https://fluxcd.control-plane.io/operator/resourcesets/introduction/) +- [PoliNetwork-CD Repository](https://github.com/polinetworkorg/polinetwork-cd) diff --git a/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md b/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md index 2b18ab0..8b03f74 100644 --- a/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md +++ b/docs/infrastructure/02-getting-started/cloud-infrastructure/index.md @@ -9,49 +9,74 @@ This section explains how we organized our infrastructure. Everything regarding the deployment of apps and services is divided into two repositories: -- [polinetworkorg/terraform](https://github.com/polinetworkorg/terraform) - Contains the Terraform code to deploy the infrastructure. This defines the resources that are created on Azure, from specific things like KeyVaults, Disks, and VMs, to the Kubernetes cluster itself. -- [polinetworkorg/polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) - Our GitOps repo, contains the manifests that define deployments specific for each app or service. This repository is connected to the ArgoCD instance running on the Kubernetes cluster. +- [polinetworkorg/terraform](https://github.com/polinetworkorg/terraform) - Contains the Terraform code for the Azure resources: the `k3s01` VM, its disks and network, the Key Vaults, the managed identities, the backup storage account. +- [polinetworkorg/polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) - Our GitOps repo. It contains the Ansible playbooks that turn the VM into a K3s node, and the Kubernetes manifests of every app and service. Flux, running in the cluster, applies them. ```mermaid flowchart TD subgraph Github tf([**terraform**]) cd([**polinetwork-cd**]) + ghcr[(GHCR images)] + end + subgraph Cloudflare + edge[Tunnel + Access] end subgraph Azure - vm[VMs and other resources] - subgraph k8s[K8S cluster] - argocd[ArgoCD] - app@{ shape: processes, label: "Apps / Services" } + kv[(Key Vaults)] + blob[(Backup storage)] + subgraph vm[VM k3s01] + subgraph k3s[K3s cluster] + flux[Flux] + eso[External Secrets] + cfd[cloudflared] + traefik[Traefik] + app@{ shape: processes, label: "Apps / Services" } + end end end tf -.-> vm - tf -.-> k8s - vm <--> k8s - cd <-.-> argocd --> app + tf -.-> kv + cd -. Ansible .-> vm + cd <-.-> flux --> app + ghcr -. image digests .-> flux + kv --> eso --> app + edge <--> cfd --> traefik --> app + vm -. backups .-> blob ``` ### **Terraform: Definition and Purpose** The **Terraform** repository is responsible for managing and provisioning cloud infrastructure. In simple terms, it defines and creates the resources needed for the infrastructure, such as: -- **Networks and subnets** (Virtual Networks, Subnets, Security Groups) -- **Virtual machines** (VMs, Containers, Kubernetes Clusters) -- **Databases** (MariaDB, PostgreSQL) -- **Storage** (Blob Storage, Virtual Disks, File Shares) -- **Security services** (Identity and Access Management, Firewalls) +- **Networks and subnets** (Virtual Network, NAT Gateway for outbound traffic, Network Security Group) +- **Virtual machines** (the `k3s01` K3s node) +- **Storage** (Managed Disks for the node, Blob Storage for backups and Terraform state) +- **Security services** (Key Vaults, managed identities, workload identity federation) The code in **Terraform** describes these resources declaratively: it defines the desired state, and Terraform applies the necessary changes to reach that state. This repository is primarily used to **create and modify the cloud infrastructure** where applications will run. +Terraform stops at the Azure boundary: it creates the VM, but it doesn't configure +what runs inside it. + ### **PoliNetwork-CD: Definition and Purpose** -The **PoliNetwork-CD** (Continuous Deployment) repository, on the other hand, is dedicated to managing and updating applications once the infrastructure has been created. Specifically, it uses **GitOps** tools (in our case, **ArgoCD**, but **Flux** is also an opinionated option) to: +The **PoliNetwork-CD** (Continuous Deployment) repository picks up where Terraform stops. It has two parts: + +- **`ansible/`** configures the VM: it mounts the data disks, hardens SSH and the + firewall, installs a pinned K3s version, schedules the encrypted backups and + bootstraps Flux. You run it by hand, rarely. +- **`clusters/k3s/`, `infrastructure/` and `apps/`** describe everything inside + Kubernetes. **Flux** watches the `main` branch and applies it continuously, so you + never run `kubectl apply` by hand. + +Through Flux, the repository is used to: -- **Define application deployments** in Kubernetes (or Kustomize, Helm, etc.) -- **Automatically update applications** when code changes -- **Manage application configurations** (e.g., configuration files, environment variables) -- **Monitor the state of applications** and ensure they remain in sync with the repository +- **Define application deployments** in Kubernetes (plain manifests organized with Kustomize, or Helm charts) +- **Automatically update applications** when a new `latest` image is published +- **Manage application configurations** (environment variables, secrets pulled from Key Vault) +- **Monitor the state of applications** and ensure they remain in sync with the repository While Terraform is responsible for infrastructure, **PoliNetwork-CD** focuses on **managing the lifecycle of applications** running on that infrastructure, ensuring that code changes are applied automatically and securely. @@ -59,10 +84,23 @@ While Terraform is responsible for infrastructure, **PoliNetwork-CD** focuses on | Repository | Main Purpose | Example of Managed Resources | |------------|--------------|------------------------------| -| **Terraform** | Creates and manages cloud infrastructure | Networks, VMs, databases, storage, security | -| **PoliNetwork-CD** | Automates application deployment and updates | Deployments, configurations, GitOps updates | +| **Terraform** | Creates and manages cloud infrastructure | VM, disks, network, Key Vaults, identities | +| **PoliNetwork-CD** (`ansible/`) | Configures the VM | Disk mounts, firewall, SSH, K3s, backups | +| **PoliNetwork-CD** (Flux) | Automates application deployment and updates | Deployments, ingresses, secrets, storage | + +In summary, **Terraform builds the foundation on which applications run**, while **PoliNetwork-CD sets up the node and manages the deployment and continuous updating of applications**. Both repositories work together to ensure a scalable and efficient cloud infrastructure. + +### How a request reaches an app + +1. A visitor opens e.g. `https://api.polinetwork.org`. DNS points to Cloudflare. +2. Cloudflare sends the request through the `k3s01` tunnel to one of the two + `cloudflared` pods in the cluster. +3. `cloudflared` forwards it to **Traefik**, the K3s ingress controller. +4. Traefik picks the app from the `Ingress` whose `host` matches, and forwards the + request to that app's `Service`. -In summary, **Terraform builds the foundation on which applications run**, while **PoliNetwork-CD manages the deployment and continuous updating of applications**. Both repositories work together to ensure a scalable and efficient cloud infrastructure. +The VM never accepts inbound connections from the Internet: the tunnel is an +outbound connection, and outbound traffic leaves Azure through a NAT Gateway. @@ -77,5 +115,6 @@ In summary, **Terraform builds the foundation on which applications run**, while ### **Additional Resources** - [Kubernetes Documentation](https://kubernetes.io/docs/) +- [K3s Documentation](https://docs.k3s.io/) +- [Flux Documentation](https://fluxcd.io/flux/) - [Terraform Documentation](https://www.terraform.io/docs/) -- [ArgoCD Documentation](https://argo-cd.readthedocs.io/en/stable/) diff --git a/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/monitoring.md b/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/monitoring.md index aa1534f..a62e571 100644 --- a/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/monitoring.md +++ b/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/monitoring.md @@ -1,60 +1,59 @@ ## Monitoring -The first thing you'll need to run is + +### Alerts + +You usually don't have to go looking for problems: Prometheus evaluates alert +rules and Alertmanager sends them to our **Telegram** alert chat. Critical alerts +repeat every hour, warnings every 12 hours. Among others, you get alerted when: + +- the node is short on CPU, memory, disk space or inodes; +- a pod is crash-looping, OOM-killed or not running, or a deployment is missing replicas; +- a Flux resource, an `ExternalSecret` or a secret store is not ready; +- the Cloudflare tunnel has no connections (all public websites are down); +- a backup hasn't succeeded on schedule. + +The rules live in `apps/monitoring/prometheus/alert-rules.yml` in polinetwork-cd. + +### Dashboards + +- [Flux Web UI](https://flux.polinetwork.org): is every app in sync with Git? Why not? +- [Grafana](https://monitoring.polinetwork.org): node and container metrics from Prometheus. +- [Uptime Kuma](https://status.polinetwork.org): is each public website up? + +### From the command line + +[SSH into the node](../../setup.md#cloudflare-warp--ssh), then run + ``` kubectl get pods -A ``` -This will give you a breakdown of all the pods running in the infrastructure, this is an example: -``` -NAMESPACE NAME READY STATUS RESTARTS AGE -argocd argo-argocd-application-controller-0 1/1 Running 0 46h -argocd argo-argocd-applicationset-controller-7fdfdc786f-hzbwv 1/1 Running 0 16d -argocd argo-argocd-dex-server-654c59ffdf-mj77j 1/1 Running 0 16d -argocd argo-argocd-notifications-controller-7987fc45b5-jgxwr 1/1 Running 0 16d -argocd argo-argocd-redis-7f845dff5-97dch 1/1 Running 0 16d -argocd argo-argocd-repo-server-9c7bf9cb-zfdcj 1/1 Running 12 (25h ago) 16d -argocd argo-argocd-server-86b5d7ddcf-686kp 1/1 Running 0 46h -bot-dev bot-dev-5d95dfc74c-6kjk9 1/1 Running 1 (114m ago) 8h -bot-dev bot-dev-6b6f9f776d-8jnts 0/1 ContainerStatusUnknown 1 28h -bot-dev bot-dev-6b6f9f776d-xtf9z 0/1 ContainerStatusUnknown 1 25h -bot-dev bot-dev-85db98944b-slj7p 0/1 ContainerStatusUnknown 2 (33h ago) 46h -bot-mat bot-mat-b5c7d6c55-xs9zh 1/1 Running 0 7h37m -bot-prod bot-prod-7967fff9bf-sbj47 1/1 Running 0 8h -calico-system calico-kube-controllers-5598966f4d-xlt4b 1/1 Running 0 16d -calico-system calico-node-x4n8x 1/1 Running 3 (46h ago) 16d -calico-system calico-node-x8wbh 1/1 Running 1 (4h14m ago) 10h -calico-system calico-typha-d78cf44dc-rv2bz 1/1 Running 0 16d -kube-system cloud-node-manager-5qkc2 1/1 Running 0 10h -kube-system cloud-node-manager-v78ms 1/1 Running 0 16d -kube-system coredns-autoscaler-5589fb5654-5gx7t 1/1 Running 0 16d -kube-system coredns-b4854dd98-gk44p 1/1 Running 0 16d -kube-system coredns-b4854dd98-nmkrp 1/1 Running 0 16d -kube-system csi-azuredisk-node-l8bfw 3/3 Running 0 10h -kube-system csi-azuredisk-node-rvmlx 3/3 Running 0 16d -kube-system csi-azurefile-node-fkpgg 3/3 Running 0 10h -kube-system csi-azurefile-node-q8khz 3/3 Running 0 16d -kube-system konnectivity-agent-bc779f4c7-2gr86 1/1 Running 0 16d -kube-system konnectivity-agent-bc779f4c7-mdbmq 1/1 Running 0 16d -kube-system kube-proxy-nlvhd 1/1 Running 0 16d -kube-system kube-proxy-vbwtj 1/1 Running 0 10h -kube-system metrics-server-f77b4cd8-k9v6f 1/1 Running 0 16d -kube-system metrics-server-f77b4cd8-zkfhl 1/1 Running 0 16d -kubernetes-dashboard dashboard-metrics-scraper-6f669b9c9b-mjzfj 1/1 Running 0 33h -kubernetes-dashboard kubernetes-dashboard-758765f476-hvzdz 1/1 Running 0 33h -mariadb mariadb-67f5f7896c-bcw2r 0/1 ContainerStatusUnknown 1 2d -mariadb mariadb-67f5f7896c-ng5df 1/1 Running 0 24h -tigera-operator tigera-operator-7c546fc6d5-hjdbc 1/1 Running 9 (25h ago) 16d -``` -You see that the dev bot is struggling a bit with multiple reboots and dead pods, which is to be expected as its allocated resources are the lowest in the cluster, given the marginal importance. -You can also see that the production bots (prod and mat) are both apparently healthy and the database is fine with only 1 reboot 24h ago. -You can take a deeper look into the monitoring with ArgoCD. -Start by getting the argocd password using -``` -kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d -``` -then configure the port-forward to your localhost:8081 -``` -kubectl port-forward -n argocd :argo-argocd-server_POD_NAME: 8081:8080 +This will give you a breakdown of all the pods running in the infrastructure. +Besides one namespace per app (`web`, `backend`, `postgres`, ...), you'll see: + +| Namespace | What runs there | +|-----------|-----------------| +| `flux-system` | Flux controllers, the Flux Web UI | +| `external-secrets` | External Secrets Operator | +| `cloudflared` | The two Cloudflare Tunnel connectors | +| `local-path-storage` | The volume provisioner | +| `kube-system` | K3s components: CoreDNS, Traefik, metrics-server, node-exporter | + +Look at the `READY`, `STATUS` and `RESTARTS` columns: a pod that is not `Running`, +or whose restart count keeps growing, needs a look. Some useful follow-ups: + +```sh +# Events and the reason of the last restart +kubectl describe pod -n + +# Logs (add --previous to see the logs of the crashed container) +kubectl logs -n deploy/ + +# CPU and memory usage per pod +kubectl top pods -A + +# Apps that Flux couldn't apply +kubectl get kustomizations -n flux-system ``` -now connect to argoCD via `localhost:8081`. -Find an overview of argocd [here](https://argo-cd.readthedocs.io/en/stable/) + +Find an overview of Flux in [Flux](../../cloud-infrastructure/03-flux.md). diff --git a/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/ssh.md b/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/ssh.md index f2db1c1..62ad756 100644 --- a/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/ssh.md +++ b/docs/infrastructure/02-getting-started/k8s-basics/day-to-day/ssh.md @@ -1,8 +1,12 @@ ## SSH Normally you shouldn't SSH into a pod to execute repetitive commands, instead you should configure them in the Dockerfile of your application. -To connect into a pod's terminal you can use the `kubectl exec` command like so +To connect into a pod's terminal, first [SSH into the node](../../setup.md#cloudflare-warp--ssh), then use the `kubectl exec` command like so ``` -kubectl exec -n :pod-namespace: :pod-name: -it -- /bin/bash +kubectl exec -n :pod-namespace: -it deploy/:deployment-name: -- /bin/sh ``` -this will open a bash terminal inside the pod.
-Mind there is no `screen` installed by default but you can use `nohup` \ No newline at end of file +this will open a shell inside a pod of that deployment (use `/bin/bash` if the image has it).
+Mind there is no `screen` installed by default but you can use `nohup`. + +:::caution +Anything you change inside a pod is lost when the pod restarts, and Flux reverts changes made with `kubectl edit`. Make persistent changes in the [polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) repository. +::: diff --git a/docs/infrastructure/02-getting-started/k8s-basics/intro.md b/docs/infrastructure/02-getting-started/k8s-basics/intro.md index 09de864..3c55e5e 100644 --- a/docs/infrastructure/02-getting-started/k8s-basics/intro.md +++ b/docs/infrastructure/02-getting-started/k8s-basics/intro.md @@ -6,13 +6,20 @@ Keep in mind that the aim of this guide is not to provide a complete overview of ## Tools needed If you want to monitor or make changes to the pods: -- Azure Cli +- Cloudflare WARP and an SSH client (`kubectl` runs on the node) + +If you want to manage secrets: +- Azure CLI If you want to make changes to the infrastructure: - Terraform - [Pre-Commits](https://github.com/antonbabenko/pre-commit-terraform) +- Ansible, only for changes to the node itself ## Configure the tools +- Connect WARP to PoliNetwork's Cloudflare Zero Trust organization +- SSH into the node with ```ssh pnadmin@10.43.1.4``` +- Run kubectl as ```k3s kubectl``` (or add the alias ```alias kubectl='k3s kubectl'```) - Login in Azure CLI using az login -- Install Kubectl and Kubelogin by running ```sudo az aks install-cli``` -- Configure Kubectl using ```az aks get-credentials --resource-group rg-polinetwork --name aks-polinetwork``` + +The full procedure is in [Setup](../setup.md). diff --git a/docs/infrastructure/02-getting-started/pipelines/cd.md b/docs/infrastructure/02-getting-started/pipelines/cd.md index 53446ea..6f77058 100644 --- a/docs/infrastructure/02-getting-started/pipelines/cd.md +++ b/docs/infrastructure/02-getting-started/pipelines/cd.md @@ -2,9 +2,15 @@ sidebar_position: 2 --- ## Continuous Delivery -Following the [Bot Pipeline](https://github.com/PoliNetworkOrg/PoliNetworkBot_CSharp/blob/master/.github/workflows/ci-cd.yaml) we deploy the built Docker Image in our [Docker Hub registry](https://hub.docker.com/u/polinetwork). -We currently have 2 environments, dev and prod on the dev and main branch respectively. -Finally we update [these](https://github.com/polinetworkorg/polinetwork-cd) k8s (Kubernetes) deployment files. +Each app repository builds its Docker image and publishes it to [GHCR](https://github.com/orgs/PoliNetworkOrg/packages) with the `latest` tag (see [CI](./ci.md)). +That's where the app pipeline stops: it doesn't touch the [polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) repository. + ## Continuous Deployment -We rely on ArgoCD to deploy our pods following the manifest present in the polinetwork-cd repository, as specified [here in the terraform deployment code](https://github.com/PoliNetworkOrg/terraform/blob/main/argocd-applications.yaml). -ArgoCD works by ensuring 0 downtime (by monitoring continuously all the deployed pods) and checking for changes on the polinetwork-cd repository (which is updated by the various pipeline whenever a change is pushed to the registry) ensuring fast deployment of fixes and changes in dev and prod environments. +We rely on [Flux](../cloud-infrastructure/03-flux.md) to deploy our pods following the manifests in the `main` branch of polinetwork-cd. +Flux continuously compares the cluster with the repository and applies any difference, so a merged PR in polinetwork-cd reaches production within minutes. + +New images don't need a commit: when a new `latest` image is published, GitHub calls a Flux webhook, Flux resolves `latest` to the new image digest and rolls out the deployment. +If the webhook is missed, Flux checks for new images every 6 hours anyway. + +Our deployments use rolling updates with `maxUnavailable: 0`: Kubernetes starts the new pod, waits for its readiness probe, and only then stops the old one, so a broken image doesn't take the app down. +Stateful services (databases, Redis, Grafana) use the `Recreate` strategy instead, because only one pod at a time can write to their volume. diff --git a/docs/infrastructure/02-getting-started/pipelines/ci.md b/docs/infrastructure/02-getting-started/pipelines/ci.md index ebd2221..12d0786 100644 --- a/docs/infrastructure/02-getting-started/pipelines/ci.md +++ b/docs/infrastructure/02-getting-started/pipelines/ci.md @@ -5,4 +5,12 @@ title: CI ## Continuous Integration -The first step of our CI processes is the Github Action workflow, in each repository. For example the [Bot Pipeline](https://github.com/PoliNetworkOrg/PoliNetworkBot_CSharp/blob/master/.github/workflows/ci-cd.yaml) +The first step of our CI processes is the GitHub Actions workflow in each repository. For example, the [web](https://github.com/PoliNetworkOrg/web/tree/main/.github/workflows) repository has: + +- `test.yml`, which typechecks and lints every push and pull request; +- `docker.yml`, which on every push to `main` builds the Docker image for `amd64` and `arm64` and publishes it to GHCR as `ghcr.io/polinetworkorg/web:latest`. + +Our K3s node is ARM64, so the `arm64` build is required. Keep publishing a +multi-arch `latest` manifest when you set up a new repository. + +The [polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd) repository has its own CI too: the **Ansible** workflow lints the playbooks and builds every Flux path with `kustomize` on each pull request. See [K3s Node](../cloud-infrastructure/02-k3s-node.md#ci). diff --git a/docs/infrastructure/02-getting-started/setup.md b/docs/infrastructure/02-getting-started/setup.md index 135cc58..b42115b 100644 --- a/docs/infrastructure/02-getting-started/setup.md +++ b/docs/infrastructure/02-getting-started/setup.md @@ -3,57 +3,112 @@ sidebar_position: 1 --- # Setup -If it's your first time accessing the infrastructure, you'll need to setup the tools and configure them. +If it's your first time accessing the infrastructure, you'll need to set up the tools and configure them. :::caution This tutorial assumes your account has the right permissions to access the infrastructure. If not, please contact the team. +SSH access to the node is restricted to the **Direttivo** Entra ID group. ::: -## Azure CLI & Kubectl +## How access works -- Download and install Kubectl [here](https://kubernetes.io/docs/tasks/tools/install-kubectl/). +The K3s node `k3s01` has **no public IP**. Its private address is `10.43.1.4`, +inside the Azure VNet `vnet-k3s`, and the Azure firewall (NSG) denies all inbound +traffic. Nothing reaches the VM directly from the Internet: -- Download and install the Azure CLI [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). +- websites reach the cluster through a **Cloudflare Tunnel** (an outbound connection + opened by `cloudflared` pods running in the cluster); +- people reach the VM through **Cloudflare WARP**, which routes `10.43.1.4` through + the same tunnel, and **Cloudflare Access for Infrastructure**, which signs a + short-lived SSH certificate after you log in with your PoliNetwork account. -- Run ```az login``` to login. +There is no kubeconfig to download: you SSH into the node and run `kubectl` there. -- Run ```sudo az aks install-cli``` to install kubectl +## Cloudflare WARP & SSH -- Run ```az aks get-credentials --resource-group rg-polinetwork --name aks-polinetwork``` to configure Kubectl. +- Install the [Cloudflare WARP client](https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/download-warp/). -- Run ```kubelogin convert-kubeconfig``` if you have installed Kubectl > v1.26 +- Enroll it in PoliNetwork's Cloudflare Zero Trust organization + (*Preferences → Account → Login to Cloudflare Zero Trust*) and log in with your + PoliNetwork account. Ask the team for the organization (team) name. -- To check if everything is working, run ```kubectl get pods -n ```. You should see a list of pods. +- With WARP connected, SSH into the node: -:::info -The possible values of `````` may vary depending on the project you're working on. If you're not sure, ask the team. + ```sh + ssh pnadmin@10.43.1.4 + ``` + + The first time, your browser asks you to authenticate. You don't need an SSH key: + Cloudflare issues a short-lived certificate for the `pnadmin` user, and the node + only accepts keys and Cloudflare Access certificates (no passwords, no root login). + +- To check if everything is working, run `k3s kubectl get pods -n `. You should see a list of pods. + +:::tip +`pnadmin` is in the `k3s-admin` group, which can read the K3s kubeconfig, so +`k3s kubectl` works without `sudo`. Add an alias to save some typing: + +```sh +echo "alias kubectl='k3s kubectl'" >> ~/.bashrc +``` + +The rest of the docs write `kubectl` and assume this alias. ::: -:::danger -Starting from Kubectl v1.26, the Azure auth plugin has been removed. If you're using a new version, you'll need to install Kubectl and Kubelogin. Then run ```kubectl get pods -n ``` again. +:::info +The possible values of `````` match the folders in +[`apps/`](https://github.com/polinetworkorg/polinetwork-cd/tree/main/apps) in the +polinetwork-cd repository (e.g. `web`, `backend`, `postgres`). Run +`kubectl get namespaces` to see them all. +::: -For now as there is no install instruction on the kubelogin repo for linux, download the latest release for your computer architecture from [here](https://github.com/Azure/kubelogin/releases), unzip it and run it with `./kubelogin convert-kubeconfig`. Follow the indicated steps +:::danger Break-glass access +If Cloudflare is down or the tunnel is broken, you can still reach the VM through +**Azure Run Command** or the **Serial Console** in the Azure portal (VM `k3s01`, +resource group `rg-polinetwork`). Use them only for emergencies. ::: -## Port Forwarding +## Azure CLI + +You still need the Azure CLI to manage secrets in Key Vault and to inspect Azure +resources. + +- Download and install the Azure CLI [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). + +- Run ```az login``` to login. -To access services running on the cluster (such as the MariaDB database), you'll need to forward the correct ports. To do so, we need the correct pod name. +You don't need `az aks get-credentials` or `kubelogin` anymore: those were only +needed for the old AKS cluster. -- Run ```kubectl get pods -n ``` to get the pod name. The pod name is listed under the ```NAME``` column. +## Web dashboards -- Run ```kubectl port-forward : -n ``` -For example, if you want to access the MariaDB database, you'll need to run +| URL | What | Who can log in | +|-----|------|----------------| +| [flux.polinetwork.org](https://flux.polinetwork.org) | Flux Web UI: sync status of every app, reconcile/suspend/resume | Direttivo, via auth.polinetwork.org | +| [monitoring.polinetwork.org](https://monitoring.polinetwork.org) | Grafana dashboards (Prometheus metrics) | Grafana accounts | +| [status.polinetwork.org](https://status.polinetwork.org) | Uptime Kuma status page | Public | + +## Port Forwarding + +To access services running on the cluster (such as the PostgreSQL database), you +need two hops: `kubectl port-forward` on the node, and an SSH tunnel from your +machine to the node. You can do both with one command: ```sh -kubectl port-forward 3306:3306 -n +ssh -L :localhost: pnadmin@10.43.1.4 \ + k3s kubectl port-forward -n svc/ : ``` -:::tip -If you're using the MariaDB database, you can use the following command to get the pod name and forward the port in one go: +For example, to reach PostgreSQL on `localhost:5432`: ```sh -PODNAME=$(kubectl get pods --no-headers -n mariadb -o custom-columns=":metadata.name" | tail -1); kubectl port-forward -n mariadb $PODNAME 3306:3306 +ssh -L 5432:localhost:5432 pnadmin@10.43.1.4 \ + k3s kubectl port-forward -n postgres svc/postgres-service 5432:5432 ``` +Keep the command running while you use the connection, and stop it with `Ctrl+C`. + +:::tip +Run `kubectl get svc -n ` on the node to find service names and ports. You can use the same template to connect to other services. ::: diff --git a/docs/infrastructure/03-Guides/add-a-new-secret.md b/docs/infrastructure/03-Guides/add-a-new-secret.md index 3023634..36de67c 100644 --- a/docs/infrastructure/03-Guides/add-a-new-secret.md +++ b/docs/infrastructure/03-Guides/add-a-new-secret.md @@ -5,336 +5,261 @@ title: Add a New Secret ## Introduction Secrets are used to configure deployments with **sensitive** data like database login, API keys, and so on. -We have deployed a "Azure Key vault" (the "KV") named "kv-polinetwork" that stores all the secrets. -Next, we can pass a secret from the KV to the k8s pod using a `SecretProviderClass`. +Secrets live in **Azure Key Vault** and never in Git. The +[External Secrets Operator](https://external-secrets.io/) (ESO), running in the +cluster, copies them into regular Kubernetes `Secret`s that your pods can use. -The following sections explain how to add a new secret. +We have two Key Vaults, one per trust boundary: -## Basics - -There are three required steps: +| Key Vault | `ClusterSecretStore` | Contains | Namespaces allowed to read it | +|-----------|---------------------|----------|-------------------------------| +| `kv-pn-apps` | `kv-pn-apps` | Application secrets (DB passwords, API keys, OAuth clients, ...) | The app namespaces listed in the store | +| `kv-pn-infra` | `kv-pn-infra` | Platform secrets (tunnel token, Flux webhook token, backup key, ...) | `cloudflared`, `flux-system` | -1. Add the secret into the "KV" -2. Create and configure a `SecretProviderClass` -3. Mount the secret volume inside the pod +ESO reads Key Vault through workload identity federation: no Azure password or +key is stored in the cluster. :::tip -Whenever there is `` in the code or in a command, that's a paramter you need to set. +Whenever there is `` in the code or in a command, that's a parameter you need to set. ::: -### Add the secret into the "KV" - -See [this section](#add-or-update-a-secret). +## Basics -### Create the `SecretProviderClass` +There are three required steps: -References: +1. Add the secret into the Key Vault +2. Declare an `ExternalSecret` in the app folder +3. Use the Kubernetes secret in the pod -- [Microsoft Guide](https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access?tabs=azure-portal&pivots=access-with-a-user-assigned-managed-identity) -- [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io) +All the YAML lives in the app folder `apps//` of +[polinetwork-cd](https://github.com/polinetworkorg/polinetwork-cd). -:::important -Check if the namespace already contains a `SecretProviderClass` before creating a new one. -Running the following command: +### Add the secret into the Key Vault -```sh -kubectl get secretproviderclass --namespace -``` +See [this section](#add-or-update-a-secret). -you should see a message like `No resources found in namespace.` +:::important Naming convention +Prefix the Key Vault secret name with the app that owns it, e.g. `backend-encryption-key`. +Key Vault names allow only letters, digits and hyphens. +::: -Otherwise the `SecretProviderClass` already exists and you can directly -[add a new secret](#add-kv-secrets-into-the-secretproviderclass). +### Declare the `ExternalSecret` -::: +Check if the app folder already contains an `external-secret.yaml`. If it does, +just [add a new entry](#add-a-key-to-an-existing-externalsecret). -This is a basic `SecretProviderClass` manifest: +Otherwise create it: -```yaml title="spc.yaml" -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass +```yaml title="apps//external-secret.yaml" +apiVersion: external-secrets.io/v1 +kind: ExternalSecret metadata: - name: # a recommended name convention is "-spc" - namespace: # if not specified, the default k8s namespace is "default" + name: azure-kv # name of the ExternalSecret spec: - provider: azure - parameters: - usePodIdentity: 'false' - useVMManagedIdentity: 'true' # Set to true for using managed identity - userAssignedIdentityID: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' # Set the clientID of the managed identity to use - tenantId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' # The tenant ID of the key vault - keyvaultName: 'kv-polinetwork' # Set to the name of your key vault + refreshInterval: 1h # how often ESO re-reads Key Vault + secretStoreRef: + kind: ClusterSecretStore + name: kv-pn-apps + target: + name: azure-kv # name of the Kubernetes Secret ESO creates + creationPolicy: Owner + deletionPolicy: Retain # keep the Secret if Key Vault is unreachable + data: + - secretKey: # key inside the Kubernetes Secret + remoteRef: + key: # secret name in Key Vault ``` -A few things to note: +and add it to the app's `kustomization.yaml`: -- `usePodIdentity` should be set to `false` and `useVMManagedIdentity` should be set to `true` since we are using a VM managed identity. -- `userAssignedIdentityID` should be set to the client ID of the managed identity we linked to the key vault. -- `tenantId` should be set to the tenant ID of our Azure tenant. - -:::tip -If you are unsure about what the `userAssignedIdentityID` and `tenantId` are, you probably aren't one of the heads of IT in PoliNetwork. -Changes to such parts of the manifest should only be done by someone with the right permissions. +```yaml title="apps//kustomization.yaml" +resources: + - namespace.yaml + # add-highlight-next-line + - external-secret.yaml + - deployment.yaml +``` -Anyway here's how to get them: +There is no need to set the namespace in the `ExternalSecret`: Kustomize sets it +from the `namespace` field of `kustomization.yaml`. -- `tenantId` +:::info +By convention, our apps call both the `ExternalSecret` and the resulting Secret +`azure-kv`, a name inherited from the AKS setup. +::: -```sh -az account show --query tenantId --output tsv -``` +### Allow the namespace to use the store -- `userAssignedIdentityID` +A `ClusterSecretStore` only serves the namespaces listed in its `conditions`. +If your app is new, add its namespace to the `kv-pn-apps` store: -```sh -az aks show --resource-group --name --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv +```yaml title="infrastructure/secret-stores/cluster-secret-stores.yaml" +apiVersion: external-secrets.io/v1 +kind: ClusterSecretStore +metadata: + name: kv-pn-apps +spec: + conditions: + - namespaces: + - auth + - backend + # add-highlight-next-line + - + ... ``` -`` and `` can be found both in our [Terraform](https://github.com/PoliNetworkOrg/terraform/) or in the Azure Portal. +Skip this and the `ExternalSecret` stays `NotReady` with an error saying the +store doesn't allow the namespace. +:::danger +Don't add app namespaces to `kv-pn-infra`. It holds the secrets that run the +platform itself. ::: -### Add "KV" secrets into the `SecretProviderClass` +### Add a key to an existing `ExternalSecret` -Inside the `SecretProviderClass` manifest, add "KV" secrets: +Append an entry under `data`: -```yaml title="spc.yaml" -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass -metadata: - name: - namespace: -spec: - provider: azure - parameters: - usePodIdentity: 'false' - useVMManagedIdentity: 'true' - userAssignedIdentityID: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - tenantId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - keyvaultName: 'kv-polinetwork' +```yaml title="apps//external-secret.yaml" + data: + - secretKey: db-pass + remoteRef: + key: postgres-root-password # add-highlight-start - objects: | - array: - - | - objectName: - objectType: secret - - | - objectName: - objectType: secret + - secretKey: + remoteRef: + key: # add-highlight-end ``` -In this example, we map two secrets (`` and ``) -from the key vault `kv-polinetwork` to a volume you can mount in your pod. - -### Mount the secret volume inside the pod - -The secret must be mounted as a volume. -Here is an example pod manifest: +## Use the secret in a pod + +### As an environment variable + +```yaml title="apps//deployment.yaml" + containers: + - name: + image: + # add-highlight-start + env: + - name: EXAMPLE_SECRET # env variable name + valueFrom: + secretKeyRef: + name: azure-kv # Kubernetes Secret name (target.name) + key: # secretKey in the ExternalSecret + # add-highlight-end +``` -```yaml title="my-spc-example-pod.yaml" -kind: Pod -apiVersion: v1 -metadata: - name: my-spc-example-pod - namespace: -spec: - containers: - - name: busybox - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 - command: - - '/bin/sleep' - - '10000' +### As a file + +```yaml title="apps//deployment.yaml" + spec: + containers: + - name: + image: + # add-highlight-start + volumeMounts: + - name: secrets + mountPath: /mnt/secrets # one file per key in this directory + readOnly: true + # add-highlight-end # add-highlight-start - volumeMounts: - - name: secrets-store # Name of the volume defined below - mountPath: '/mnt/secrets-store' # Path where the secrets are mounted inside the pod - readOnly: true # A secret should be mounted as read-only + volumes: + - name: secrets + secret: + secretName: azure-kv # add-highlight-end - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 250m - memory: 256Mi - -# add-highlight-start - volumes: - - name: secrets-store # Name of the volume - it can be whatever you want - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: '-spc' # The name of the SecretProviderClass -# add-highlight-end ``` -After applying this manifest to the cluster, you can retrieve the secret by reading the files in the `/mnt/secrets-store/` directory (`mountPath` parameter). -To test if everything works, try running the following command. +Unlike the old CSI driver, you don't need to mount a volume to use a secret as +an environment variable: pick whichever form the app needs. -```sh -kubectl exec -it my-spc-example-pod -- cat /mnt/secrets-store/ -``` +## Deploy and check -:::tip -More than one secret can be mounted at the same time by adding more entries in the array, -all of them can be found in the same directory, you can see them by running: +Open a PR into polinetwork-cd. Once it's merged, Flux applies it. Then, from the node: ```sh -kubectl exec -it my-spc-example-pod -- ls /mnt/secrets-store -``` +# Ready=True and SecretSynced means ESO copied the secret +kubectl get externalsecret -n -In the [previous section](#add-kv-secrets-into-the-secretproviderclass), we have added `` and ``. - -::: - -## Loading the secret inside an Environment Variable - -:::important -Even if you want to load the secret as an ENV variable, it's **REQUIRED** to follow every steps in the previous section, including -[mounting the secret volume](#mount-the-secret-volume-inside-the-pod). +# The keys in the Kubernetes Secret (values are not printed) +kubectl describe secret azure-kv -n +``` -You can find more about why this is the case [here](https://github.com/kubernetes-sigs/secrets-store-csi-driver/issues/813) -::: +## Updating a secret value -To load the secret as an Environment Variable follows the following steps. +ESO re-reads Key Vault every `refreshInterval` (1 hour). To pick up a new value +immediately, force a sync: -### Register the secret as a k8s secret +```sh +kubectl annotate externalsecret azure-kv -n \ + force-sync="$(date +%s)" --overwrite +``` -Inside the `ServiceProviderClass` manifest, add a new field `secretObjects` to create a -new k8s secret collection: +Environment variables are read only when a container starts, so restart the app: -```yaml title="spc.yaml" -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass -metadata: - name: - namespace: -spec: - provider: azure - # add-highlight-start - secretObjects: - # each of these is a COLLECTION of secrets. - # multiple separate collections can be defined, but to identify Azure secrets from other k8s secrets, - # we use a collection that contains multiple secrets (as if it were an Object in fact). - # important: it only needs to be created the first time, then just add a key underneath - - secretName: azure-kv - type: Opaque - data: - # the secret that we want to expose also as k8s secret should be added here. - # important to distinguish objectName (reference to "KV") from key (custom name) - - objectName: # secret name inside the "KV" - key: example-secret # custom k8s secret's key - # add-highlight-end - parameters: - usePodIdentity: 'false' - useVMManagedIdentity: 'true' - userAssignedIdentityID: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - tenantId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - keyvaultName: 'kv-polinetwork' - objects: | - array: - - | - objectName: - objectType: secret +```sh +kubectl rollout restart deployment/ -n ``` -### Use the secret as environment variable - -After [mounting the secret volume](#mount-the-secret-volume-inside-the-pod) and [configuring the k8s secret](#register-the-secret-as-a-k8s-secret), -you can add an environment variable with the secret as value reference: +Secrets mounted as files are updated in place after a short delay, but your app +has to re-read them. -```yaml title="my-spc-example-pod.yaml" -kind: Pod -apiVersion: v1 -metadata: - name: my-spc-example-pod - namespace: -spec: - containers: - - name: busybox - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 - command: - - '/bin/sleep' - - '10000' - volumeMounts: - - name: secrets-store - mountPath: '/mnt/secrets-store' - readOnly: true - # add-highlight-start - env: - - name: EXAMPLE_SECRET # env variable name (independent from the name of the secret) - valueFrom: - secretKeyRef: - name: azure-kv # k8s secret collection name - key: example-secret # secret key inside of k8s secret collection specified in the line above - # add-highlight-end - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 250m - memory: 256Mi - - volumes: - - name: secrets-store - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: '-spc' -``` +## Key Vault management -## "KV" management +You need the Azure CLI, logged in with an account that has the +**Key Vault Secrets Officer** role on the vault. ### Add or update a secret You can add (or update) a secret with the following command: ```sh -az keyvault secret set --vault-name "kv-polinetwork" --name "" --value "" +az keyvault secret set --vault-name "kv-pn-apps" --name "" --value "" ``` :::tip -If the secret value is too long or you don't want to copy/paste in terminal you can use a file instead. +If the secret value is too long or you don't want to copy/paste in terminal (and +leave it in your shell history) you can use a file instead. 1. Paste the secret value into a text file (no extension or ".txt" is valid) on a **single line** 2. Run the following command ```sh - az keyvault secret set --vault-name "kv-polinetwork" --name "" --file "" + az keyvault secret set --vault-name "kv-pn-apps" --name "" --file "" ``` +3. Delete the file + ::: ### Delete a secret -You can delete a secret with the following command: +Remove it from every `ExternalSecret` first, otherwise the `ExternalSecret` turns +`NotReady`. Then delete it with: ```sh -az keyvault secret delete --vault-name "kv-polinetwork" --name "" +az keyvault secret delete --vault-name "kv-pn-apps" --name "" ``` -After the deletion, the secret remains in a `Recoverable State` for 7 days. +After the deletion, the secret remains in a `Recoverable State` for **90 days**. During this period the secret can be recovered with the following command: ```sh -az keyvault secret recover --vault-name "kv-polinetwork" --name "" +az keyvault secret recover --vault-name "kv-pn-apps" --name "" ``` -To **permanently delete** that secret, after 7 days from deletion you can run the following command: - -```sh -az keyvault secret purge --vault-name "kv-polinetwork" --name "" -``` +:::warning +Both vaults have **purge protection** enabled: a deleted secret can't be purged +before its 90 days are over, and its name can't be reused in the meantime unless +you recover it. +::: :::note -To **update/re-set** that secret, during the 7 days retention period, you must recover it first, +To **update/re-set** that secret, during the 90 days retention period, you must recover it first, then run the command to update it -If you try to `set` a secret that has been deleted within the last 7 days without restoring it first, you get the following error: +If you try to `set` a secret that has been deleted within the last 90 days without restoring it first, you get the following error: ```sh #error-highlight-start @@ -349,140 +274,72 @@ Inner error: { ::: -## Examples - -In this section you can find working examples. -It is recommended to read the ["Basics" section](#basics) to understand how it works. - -:::note -You still need to set `tenantId`, `userAssignedIdentityID` and secret keys to make it work. -::: +## Example -### Minimal +A complete app that reads one secret, `my-app-api-key`, from `kv-pn-apps`: -```yaml title="test-secret/spc.yaml" showLineNumbers -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass +```yaml title="apps/my-app/external-secret.yaml" showLineNumbers +apiVersion: external-secrets.io/v1 +kind: ExternalSecret metadata: - name: test-secret-spc - namespace: test-secret + name: azure-kv spec: - provider: azure - parameters: - usePodIdentity: 'false' - useVMManagedIdentity: 'true' - userAssignedIdentityID: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - tenantId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - keyvaultName: 'kv-polinetwork' - objects: | - array: - - | - objectName: - objectType: secret - - | - objectName: - objectType: secret + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: kv-pn-apps + target: + name: azure-kv + creationPolicy: Owner + deletionPolicy: Retain + data: + - secretKey: api-key + remoteRef: + key: my-app-api-key ``` -```yaml title="test-secret/example-pod.yaml" showLineNumbers -kind: Pod -apiVersion: v1 +```yaml title="apps/my-app/deployment.yaml" showLineNumbers +apiVersion: apps/v1 +kind: Deployment metadata: - name: example-pod - namespace: test-secret + name: my-app spec: - containers: - - name: busybox - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 - command: - - '/bin/sleep' - - '10000' - volumeMounts: - - name: secrets-store - mountPath: '/mnt/secrets-store' - readOnly: true - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 250m - memory: 256Mi - - volumes: - - name: secrets-store - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: 'test-secret-spc' + replicas: 1 + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: my-app + image: ghcr.io/polinetworkorg/my-app:latest + env: + - name: API_KEY + valueFrom: + secretKeyRef: + name: azure-kv + key: api-key + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + memory: 128Mi ``` -### Environment Variable +```yaml title="apps/my-app/kustomization.yaml" showLineNumbers +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization -```yaml title="test-secret-env/spc.yaml" showLineNumbers -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass -metadata: - name: test-secret-env-spc - namespace: test-secret-env -spec: - provider: azure - secretObjects: - - secretName: azure-kv - type: Opaque - data: - - objectName: - key: example-secret - parameters: - usePodIdentity: 'false' - useVMManagedIdentity: 'true' - userAssignedIdentityID: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - tenantId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - keyvaultName: 'kv-polinetwork' - objects: | - array: - - | - objectName: - objectType: secret -``` +namespace: my-app -```yaml title="test-secret-env/example-pod.yaml" showLineNumbers -kind: Pod -apiVersion: v1 -metadata: - name: example-pod - namespace: test-secret-env -spec: - containers: - - name: busybox - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 - command: - - '/bin/sleep' - - '10000' - volumeMounts: - - name: secrets-store - mountPath: '/mnt/secrets-store' - readOnly: true - env: - - name: EXAMPLE_SECRET - valueFrom: - secretKeyRef: - name: azure-kv - key: example-secret - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 250m - memory: 256Mi - - volumes: - - name: secrets-store - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: 'test-secret-env-spc' +resources: + - namespace.yaml + - external-secret.yaml + - deployment.yaml ``` + +Plus `my-app` in the `kv-pn-apps` store conditions, as shown [above](#allow-the-namespace-to-use-the-store). diff --git a/docs/infrastructure/03-Guides/add-storage.md b/docs/infrastructure/03-Guides/add-storage.md index 874c8f8..787e8a5 100644 --- a/docs/infrastructure/03-Guides/add-storage.md +++ b/docs/infrastructure/03-Guides/add-storage.md @@ -5,76 +5,128 @@ title: Add Storage ## Core Concepts Many applications and services need a physical volume or storage to store their data or configuration. -As you should know, we use AKS (Kubernetes) to host everything from databases to telegram bots. -In our AKS cluster we have (ATTOW) a single node (VM) `Standard_B2ms` which allows to attach only 4 disks, -one of which is the VM's disk itself. -This means that you can only attach 3 managed disks to the Cluster, resulting in limited deployable services. +On K3s, volumes are plain directories on the node's data disks, created on demand by the +[local-path provisioner](https://github.com/rancher/local-path-provisioner) when a +[PVC](https://kubernetes.io/docs/concepts/storage/persistent-volumes) asks for one. +We replaced Longhorn and the Azure Managed Disk volumes we used on AKS with it. -To solve this problem, we added [Longhorn](https://longhorn.io/) so that we can create multiple -[PVC](https://kubernetes.io/docs/concepts/storage/persistent-volumes) from the VM's disk -(ATTOW it's a 128GB Standard SSD). +There are two storage classes, one per data disk: + +| Storage class | Disk | Path on the node | Use it for | +|---------------|------|------------------|------------| +| `fast` | Premium SSD v2, 64 GB | `/srv/fast/volumes///` | Latency-sensitive databases (today: PostgreSQL and the Redis instances) | +| `standard` (default) | Standard SSD, 128 GB | `/srv/standard/volumes///` | Everything else (today: InfluxDB, Prometheus, Grafana, Uptime Kuma) | :::important -Longhorn is a handy tool to create multiple PVC/volumes without creating new disks, but this comes with degraded performance and stability. -If you want to deploy something critical like a production database, it's recommended to create a dedicated -[Managed Disk](https://learn.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview). +- **The size is not enforced.** `storage: 4Gi` in a PVC is a label, not a quota: a volume can + grow until the whole disk is full, and then every app on that disk breaks. + Check the free space first, and keep an eye on the `PersistentVolumeHigh` and + `NodeDiskHigh` alerts. +- **Volumes are `ReadWriteOnce`** and live on one node: only one pod at a time should + write to them. +- **Volumes are retained.** Deleting a PVC doesn't delete its directory (`reclaimPolicy: Retain`). ::: -## Add a Longhorn volume +## Add a volume -First of all, you must login in [our Longhorn dashboard](https://longhorn.polinetwork.org) and check -how much space is available for scheduling a new volume. -:::tip -- `Schedulable` is space you can use to create new volumes -- `Reserved` is space reserved to the VM, not available to create new volumes +First of all, [SSH into the node](../getting-started/setup#cloudflare-warp--ssh) and check +how much space is available on the disk you want to use: -How much `Reserved` space is configured in the `Node` page, but at least 20-25 GB is recommended. -::: +```sh +df -h /srv/fast /srv/standard +``` -Once you verified you have sufficient `Schedulable` space available, you can create the PVC in the k8s manifest: +Once you verified you have sufficient space available, add the PVC to the app manifest: -```yaml title="my-longhorn-example.yaml" +```yaml title="apps//deployment.yaml" # add-highlight-start apiVersion: v1 kind: PersistentVolumeClaim metadata: name: -pvc + annotations: + kustomize.toolkit.fluxcd.io/prune: disabled # never let Flux delete the data spec: accessModes: - ReadWriteOnce - storageClassName: longhorn-static + storageClassName: resources: requests: storage: # add-highlight-end --- -kind: Pod -apiVersion: v1 +apiVersion: apps/v1 +kind: Deployment metadata: - name: my-longhorn-example-pod - namespace: + name: spec: - containers: - - name: busybox - image: registry.k8s.io/e2e-test-images/busybox:1.29-4 - command: - - '/bin/sleep' - - '10000' + replicas: 1 + # add-highlight-start + strategy: + type: Recreate # stop the old pod before starting the new one + # add-highlight-end + selector: + matchLabels: + app: + template: + metadata: + labels: + app: + spec: + containers: + - name: + image: + # add-highlight-start + volumeMounts: + - name: data # Name of the volume defined below + mountPath: /app/data # Where you need to mount the volume + # add-highlight-end # add-highlight-start - volumeMounts: - - name: my-volume # Name of the volume defined below - mountPath: /app/data # Where you need to mount the volume + volumes: + - name: data # it can be whatever you want + persistentVolumeClaim: + claimName: -pvc # the name of the PVC defined above # add-highlight-end -# add-highlight-start - volumes: - - name: my-volume # it can be whatever you want - persistentVolumeClaim: - claimName: -pvc # the name of the PVC defined above -# add-highlight-end ``` Parameters: -- `namespace` k8s namespace +- `namespace` k8s namespace (set by the app's `kustomization.yaml`) - `name` should identify the app where the volume will be used -- `size` size of the volume. Check [resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory). +- `storage-class` `fast` or `standard` (see [the table above](#core-concepts)) +- `size` expected size of the volume. Check [resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory). e.g. `300M`, `1Gi`, `200Mi` + +:::warning Both are required +- The `kustomize.toolkit.fluxcd.io/prune: disabled` annotation keeps Flux from deleting the + PVC if it disappears from Git (e.g. a bad refactor). CI rejects PVCs without it. +- The `Recreate` strategy avoids two pods writing to the same volume during a rollout. +::: + +The volume is created when the first pod using it is scheduled, so the PVC stays +`Pending` until then: that's expected. + +:::tip File permissions +local-path directories are owned by `root`. If your container runs as a non-root +user, set `securityContext.fsGroup` on the pod to the user's group (see `apps/redis` +for an example). +::: + +## Back it up + +Volumes are **not** backed up automatically. The application backup on the node +only includes the databases and directories it knows about. If the new volume +holds data we can't afford to lose, add it to the backup script: see +[Backups](./backups.md#adding-a-new-app). + +## Inspecting a volume + +Since a volume is just a directory, you can look at it from the node: + +```sh +sudo ls -la /srv//volumes//-pvc/ +``` + +:::danger +Don't edit files of a running database from the node. Stop the app first, or go +through the app itself. +::: diff --git a/docs/infrastructure/03-Guides/backups.md b/docs/infrastructure/03-Guides/backups.md new file mode 100644 index 0000000..ea75fd4 --- /dev/null +++ b/docs/infrastructure/03-Guides/backups.md @@ -0,0 +1,154 @@ +--- +title: Backups +--- + +## Overview + +Two systemd timers on the node, installed by Ansible (`ansible/roles/backup`), back up the cluster: + +| Backup | When (UTC) | Contents | +|--------|------------|----------| +| **Applications** (`k3s-application-backup`) | Every 6 hours (00:45, 06:45, 12:45, 18:45) | PostgreSQL (`pg_dumpall`, all databases and roles), Redis (`redis` and `bot-ts` RDB snapshots), InfluxDB (`influx backup`), Uptime Kuma and Grafana (SQLite databases and files) | +| **Control plane** (`k3s-control-plane-backup`) | Daily at 02:15 | The K3s datastore (`state.db`), the K3s server token and `config.yaml` | + +Each run: + +1. takes consistent online copies (no app is stopped); +2. packs them in a `.tar.gz` with a `SHA256SUMS` file and encrypts it with + [age](https://age-encryption.org/); +3. uploads it to the private `backups` container of the `polinetworkbackups` + storage account, using the `id-k3s-backup` managed identity; +4. publishes its success time to Prometheus. + +:::important Encryption +The node only has the age **public** key. The private key (the "identity") is the +secret `backup-age-identity` in the `kv-pn-infra` Key Vault, and it must never be +copied to the node. Without it, the backups can't be read. +::: + +## Retention + +| Where | Path | Kept for | +|-------|------|----------| +| Node | `/srv/standard/backups/k3s-applications/`, `/srv/standard/backups/k3s-control-plane/` | 7 days | +| Blob Storage | `backups/k3s-applications/k3s01/`, `backups/k3s-control-plane/k3s01/` | 90 days, immutable for the first 30 | + +The storage account only accepts traffic from the K3s subnet, so download backups from the node. + +## Monitoring + +The `BackupStale` alert fires on Telegram when there has been no successful +application backup for 7 hours or no control-plane backup for 26 hours. +`BackupMetricsMissing` fires if the metrics disappear altogether. + +From the node: + +```sh +# Next and last runs +systemctl list-timers 'k3s-*-backup.timer' + +# Logs of the last run +sudo journalctl -u k3s-application-backup.service -n 50 + +# Run a backup now +sudo systemctl start k3s-application-backup.service +``` + +## Adding a new app + +Volumes are not backed up automatically. If you add an app with data worth keeping +(see [Add Storage](./add-storage.md)), add it to +`ansible/roles/backup/templates/k3s-application-backup.sh.j2` in polinetwork-cd, using +the right tool for the data: + +- a database: its own dump/backup command, through `kubectl exec` (like PostgreSQL and InfluxDB); +- an SQLite file: the `sqlite_backup` helper, which takes a consistent copy and archives + the other files in the volume; +- Redis: the `redis_snapshot` helper. + +Then run the playbook (see [K3s Node](../02-getting-started/cloud-infrastructure/02-k3s-node.md#running-the-playbooks)): +`verify.yml` takes a fresh backup and checks it. + +## Restoring + +:::danger +Restoring overwrites live data. Agree on it with the Direttivo first, and take a +fresh backup right before (`sudo systemctl start k3s-application-backup.service`) +so you can go back. +::: + +### 1. Get the archive + +The last 7 days are on the node, in `/srv/standard/backups/k3s-applications/`, +readable only by root. Copy the one you need to your home: + +```sh +sudo cp /srv/standard/backups/k3s-applications/.tar.gz.age ~ +sudo chown pnadmin ~/.tar.gz.age +``` + +For older backups, list and download them from Blob Storage on the node, with the +backup identity (its client ID is `backup_identity_client_id` in `ansible/vars/main.yml`): + +```sh +TOKEN=$(curl -s -H Metadata:true \ + "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com%2F&client_id=" \ + | jq -r .access_token) + +# List +curl -s -H "Authorization: Bearer $TOKEN" -H "x-ms-version: 2023-11-03" \ + "https://polinetworkbackups.blob.core.windows.net/backups?restype=container&comp=list&prefix=k3s-applications/k3s01/" \ + | grep -o '[^<]*' + +# Download +curl -s -H "Authorization: Bearer $TOKEN" -H "x-ms-version: 2023-11-03" \ + -o ~/.tar.gz.age \ + "https://polinetworkbackups.blob.core.windows.net/backups/k3s-applications/k3s01/.tar.gz.age" +``` + +### 2. Decrypt it on your machine + +Copy the encrypted archive to your machine and decrypt it there with +[age](https://github.com/FiloSottile/age#installation) and the identity from Key Vault: + +```sh +scp pnadmin@10.43.1.4:.tar.gz.age . + +az keyvault secret show --vault-name kv-pn-infra --name backup-age-identity \ + --query value -o tsv > age-identity.txt + +mkdir restore +age --decrypt --identity age-identity.txt .tar.gz.age | tar -xz -C restore +(cd restore && sha256sum -c SHA256SUMS) + +rm age-identity.txt +``` + +`restore/` now contains `postgres-dumpall.sql`, `redis.rdb`, `bot-ts-redis.rdb`, +`influxdb-backup.tar`, `uptime-kuma.db`, `grafana.db` and the `*-files.tar` archives. + +:::caution +These files hold personal data and credentials in clear text. Delete them as soon as +you're done. +::: + +### 3. Restore the data + +For PostgreSQL, the dump recreates every database (`--clean --if-exists`), so you can +stream it into the running server: + +```sh +ssh pnadmin@10.43.1.4 \ + "k3s kubectl exec -i -n postgres deploy/postgres -- sh -c 'psql --username=\"\$POSTGRES_USER\" --dbname=postgres'" \ + < restore/postgres-dumpall.sql +``` + +For the other services, stop the app (`kubectl scale --replicas=0`), put the files back +in its volume under `/srv//volumes///`, and scale it up +again. Follow the upstream docs for +[Redis](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) and +[InfluxDB](https://docs.influxdata.com/influxdb/v2/admin/backup-restore/restore/). + +The control-plane backup is only needed to rebuild the node from scratch: it restores +the K3s datastore, including the service-account signing key that Azure workload +identity trusts. diff --git a/docs/infrastructure/03-Guides/deploy-a-new-app.md b/docs/infrastructure/03-Guides/deploy-a-new-app.md index 6e6dbe8..a43c6b4 100644 --- a/docs/infrastructure/03-Guides/deploy-a-new-app.md +++ b/docs/infrastructure/03-Guides/deploy-a-new-app.md @@ -4,206 +4,365 @@ title: Deploy a New App ## Introduction -We use k8s to deploy our applications. -You can choose, based on your needs, to use basic k8s manifests, [Kustomize](https://kustomize.io/) or [Helm Charts](https://helm.sh/docs/topics/charts/). +We use k8s to deploy our applications, on our [K3s node](../02-getting-started/cloud-infrastructure/02-k3s-node.md). +Each app is a folder of plain k8s manifests in the +[polinetwork-cd](https://github.com/PoliNetworkOrg/polinetwork-cd/) repository, tied +together with [Kustomize](https://kustomize.io/). [Helm Charts](https://helm.sh/docs/topics/charts/) +are possible too, through a Flux `HelmRelease` (see `infrastructure/external-secrets` for an example). -If you have access to PoliNetwork's k8s cluster (you can perform `kubectl` commands) you can directly deploy from your machine. -Although, this is not recommended in a production environment. -Check [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Flinux%2Fx86-64%2Fstable%2Fbinary+download) to install a k8s cluster in your machine for testing purposes. +You don't deploy from your machine: you open a PR, and once it's merged into `main`, +Flux applies it. +Check [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Flinux%2Fx86-64%2Fstable%2Fbinary+download) or [K3s](https://docs.k3s.io/quick-start) to run a k8s cluster on your machine for testing purposes. :::note CD -To deploy our applications, we use [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) -with a [Git File Generator](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/#git-generator-files) -pointing to the [polinetwork-cd](https://github.com/PoliNetworkOrg/polinetwork-cd/) GitHub repo. +To deploy our applications, we use [Flux](https://fluxcd.io/flux/) syncing the +`clusters/k3s` folder of the `main` branch of +[polinetwork-cd](https://github.com/PoliNetworkOrg/polinetwork-cd/). -For more info, check out *[Getting Started / Basic Knowledge]* +For more info, check out [Flux](../02-getting-started/cloud-infrastructure/03-flux.md). ::: ## Procedure -1. Deploy a [Docker image](https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-an-image/) -2. [Create the app structure](#create-the-app-structure) -3. Open a PR into polinetwork-cd -4. Monitor deploy on ArgoCD +1. Publish a [Docker image](https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-an-image/) with an **`arm64`** variant +2. [Create the app folder](#create-the-app-folder) +3. [Register the app with Flux](#register-the-app-with-flux) +4. Optionally, [publish it on a domain](#publish-it-on-a-domain), [add secrets](./add-a-new-secret.md), [add storage](./add-storage.md) and [enable automatic image updates](#automatic-image-updates) +5. Open a PR into polinetwork-cd, wait for CI, merge +6. Monitor the deploy on the [Flux Web UI](https://flux.polinetwork.org) + +:::important ARM64 images +The node is ARM64. An image built only for `amd64` can't run there, and the pod fails +with `exec format error`. Build a multi-arch image (see the +[web workflow](https://github.com/PoliNetworkOrg/web/blob/main/.github/workflows/docker.yml) +for an example), or check that the third-party image you use publishes `linux/arm64`. +::: -### Create the app structure +### Create the app folder This is the crucial part, so make sure to follow it step-by-step. 1. Clone [polinetwork-cd](https://github.com/PoliNetworkOrg/polinetwork-cd) repo 2. Create a new branch (named as you like, e.g. `mc-server-deploy`) 3. Create this minimal file structure: ``` -🗀 -├─ 🗀 src -│ └── deployment.yaml -│ -└─ config.json +🗀 apps +└─ 🗀 + ├── kustomization.yaml + ├── namespace.yaml + └── deployment.yaml ``` :::info The `` should be a slug (lowercase words separated by hyphens) identifier of the application/service you want to deploy. -You can append, if necessary, an enviornment tag (e.g. `app-dev`, `app-prod`). -Make sure that you use the same `` value for both: -- folder name -- namespace field in k8s resources +Use the same `` value for the folder name, the `Namespace` resource and +the `namespace` field of `kustomization.yaml`. You don't need to write the namespace in +the other manifests: Kustomize adds it. ::: - -Inside `src/` you can put one of the following: -- base k8s manifest ([example](https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/controllers/nginx-deployment.yaml)) -- [Kustomize](https://kubectl.docs.kubernetes.io/guides/introduction/kustomize/) -- [Helm Chart](https://helm.sh/docs/topics/charts/) - -How these build tools are detected and distinguished is described [here](https://argo-cd.readthedocs.io/en/stable/user-guide/tool_detection/#tool-detection). -:::important -If you would like to use the Docker Image Auto-Updater, base k8s manifests are not supported, you must choose between Kustomize and Helm Chart. +```yaml title="apps//kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization -For more info, check the [relative section](#docker-image-auto-updater). -::: +namespace: -## Docker Image Auto-Updater -We use [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io/en/stable/) to -automatically update docker images in deployments. -This is an optional feature, but the alternative is to use a docker image fixed version (e.g. `nginx:1.27.4`). +resources: + - namespace.yaml + - deployment.yaml +``` -We use the `digest` strategy by default (also the only one we documented so far). -Check out [the other strategies available](https://argocd-image-updater.readthedocs.io/en/stable/basics/update-strategies/). +```yaml title="apps//namespace.yaml" +apiVersion: v1 +kind: Namespace +metadata: + name: + annotations: + kustomize.toolkit.fluxcd.io/prune: disabled +``` -As mentioned in the [section above](#create-the-app-structure), you must use Kustomize or Helm Charts. +The `prune: disabled` annotation keeps Flux from deleting the namespace, and with it +everything inside, if the app is removed from Git by mistake. -:::warning -Consider the following manifest: -```yaml -kind: Pod +```yaml title="apps//deployment.yaml" apiVersion: v1 +kind: Service metadata: - name: example-pod - namespace: test + name: -service spec: - containers: - - name: nginx - image: nginx:latest + selector: + app: + ports: + - name: http + port: + targetPort: +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: +spec: + replicas: 1 + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 0 # keep the old pod until the new one is ready + maxSurge: 1 + selector: + matchLabels: + app: + template: + metadata: + labels: + app: + spec: + containers: + - name: + image: ghcr.io/polinetworkorg/:latest + imagePullPolicy: IfNotPresent + ports: + - containerPort: + readinessProbe: + tcpSocket: + port: + periodSeconds: 5 + livenessProbe: + tcpSocket: + port: + periodSeconds: 10 + resources: + requests: + cpu: 10m + memory: 128Mi + limits: + memory: 256Mi ``` -the image would **not update** if you **don't follow** this guide, even though there is `latest`. -::: -### Digest with Kustomize -- [Full example](https://github.com/PoliNetworkOrg/polinetwork-cd/tree/main/tests/docker-test) +A few rules we follow for every app: -In this configuration, we are using a Kustomize deployment with `digest` strategy. -These are the steps to follow: (we are using `toto04/testcontainer` as the docker image) +- **Requests and limits.** Everything shares one node, so every container declares + its memory request and limit. Base the numbers on what the app actually uses + (`kubectl top pods`). +- **Probes.** The readiness probe is what makes rolling updates safe: without it, + Kubernetes kills the old pod before the new one is serving. +- **Apps with a volume** use `strategy: Recreate` instead (see [Add Storage](./add-storage.md)). +- Apps talk to each other through their Service: + `http://..svc.cluster.local:`. -1. Create a `src/kustomization.yaml` with this content: -```yaml -apiVersion: kustomize.config.k8s.io/v1beta1 +### Register the app with Flux + +Flux only deploys the apps listed in `clusters/k3s/apps/`. Create a Flux +`Kustomization` for the app: + +```yaml title="clusters/k3s/apps/.yaml" +apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization +metadata: + name: + namespace: flux-system +spec: + dependsOn: + - name: infrastructure-secret-stores + - name: infrastructure-storage + - name: infrastructure-traefik + interval: 30m + retryInterval: 1m + timeout: 5m + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./apps/ +``` + +If the app needs another app to be up first (e.g. a database), add it to +`dependsOn` (e.g. `- name: postgres`). +Then add it to the list: + +```yaml title="clusters/k3s/apps/kustomization.yaml" resources: -- deployment.yaml + - admin.yaml + ... + # add-highlight-next-line + - .yaml ``` -2. As declared in the `resources` field above, we need a k8s manifest `src/deployment.yaml`. -Create it and set the `spec.containers[x].image` field as a fixed digest, example: -```yaml +If you want automatic image updates, write a `ResourceSet` instead of this +`Kustomization`: see [below](#automatic-image-updates). + +### Publish it on a domain + +Public traffic arrives through the Cloudflare Tunnel to Traefik, which routes it by +hostname. Two things are needed. + +1. An `Ingress` in the app folder (and in `kustomization.yaml`): + +```yaml title="apps//ingress.yaml" +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: + annotations: + traefik.ingress.kubernetes.io/router.entrypoints: web spec: - containers: - - name: docker-test-pod - image: toto04/testcontainer@sha256:b5ec2efcc11e90c2fd955cd1c01f3fdf7c2f18c77c213360addcd37be7e8f2f3 + ingressClassName: traefik + rules: + - host: .polinetwork.org + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: -service + port: + number: ``` -:::tip -If you are using Docker Hub, you can find a digest going to the image homepage ([example](https://hub.docker.com/r/toto04/testcontainer/tags)) -under the tag you want to use (e.g. `latest`) -::: +2. A public hostname on the **`k3s01`** tunnel in the Cloudflare Zero Trust dashboard + (*Networks → Tunnels*), pointing to Traefik like the existing hostnames. The tunnel + is managed from the dashboard, not from Git, so ask someone in the Direttivo to add it. + +Cloudflare terminates HTTPS, so the `Ingress` only needs the plain `web` entrypoint: +no certificates in the cluster. + +## Automatic image updates +Flux can redeploy the app every time a new `latest` image is published, pinning the +image digest (see [how it works](../02-getting-started/cloud-infrastructure/03-flux.md#automatic-image-updates)). +This is an optional feature: the alternative is a fixed version or digest in +`deployment.yaml` (e.g. `nginx:1.27.4`), bumped with a PR. -3. In the `config.json` file set the image(s) to automatically update: +:::warning +Consider the following manifest: ```yaml - "image": { - "image_list": "toto04/testcontainer:latest", - "update_strategy": "digest" - } + containers: + - name: nginx + image: nginx:latest ``` +the image would **not update** if you **don't follow** this guide, even though there is `latest`: +the node already has an image called `nginx:latest` and has no reason to pull it again. +::: + +The image must be **public** on GHCR: neither Flux nor the node use registry credentials. +Using the GHCR package `ghcr.io/polinetworkorg/`: -In this configuration, the image `toto04/testcontainer` would be automatically updated following the tag `latest`. +1. Add a provider that tracks its `latest` tag: +```yaml title="infrastructure/image-automation/image-providers.yaml" +--- +apiVersion: fluxcd.controlplane.io/v1 +kind: ResourceSetInputProvider +metadata: + name: -image + namespace: flux-system + annotations: + fluxcd.controlplane.io/reconcileEvery: "6h" +spec: + type: OCIArtifactTag + url: oci://ghcr.io/polinetworkorg/ + filter: + includeTag: '^latest$' + limit: 1 +``` +2. Let the GitHub webhook trigger it: -:::tip -You can also specify multiple images to update: -```yaml title="config.json" - "image": { -# add-highlight-start - "image_list": "x/image1:latest x/image2:latest", -# add-highlight-end - "update_strategy": "digest" - } +```yaml title="infrastructure/image-automation/receiver.yaml" + resources: + ... + # add-highlight-start + - apiVersion: fluxcd.controlplane.io/v1 + kind: ResourceSetInputProvider + name: -image + # add-highlight-end ``` -```yaml title="src/deployment.yaml" -apiVersion: apps/v1 -kind: Deployment -metadata: - name: deploy-1 - namespace: -spec: - replicas: 1 - selector: - matchLabels: - app: app-1 - template: - metadata: - labels: - app: app-1 - spec: - containers: - - name: app-1-pod -# add-highlight-start - image: x/image1@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -# add-highlight-end ---- -apiVersion: apps/v1 -kind: Deployment +3. Replace `clusters/k3s/apps/.yaml` with a `ResourceSet` that renders the + same Kustomization, plus an `images` override with the digest: + +```yaml title="clusters/k3s/apps/.yaml" +apiVersion: fluxcd.controlplane.io/v1 +kind: ResourceSet metadata: - name: deploy-2 - namespace: + name: + namespace: flux-system spec: - replicas: 2 - selector: - matchLabels: - app: app-2 - template: - metadata: - labels: - app: app-2 - spec: - containers: - - name: app-2-pod -# add-highlight-start - image: x/image2@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -# add-highlight-end + inputStrategy: + name: Permute + inputsFrom: + - kind: ResourceSetInputProvider + name: -image + resources: + - apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: + namespace: flux-system + annotations: + # Never garbage-collect the app if the provider exports no inputs. + fluxcd.controlplane.io/prune: disabled + spec: + dependsOn: + - name: infrastructure-secret-stores + - name: infrastructure-storage + - name: infrastructure-traefik + interval: 30m + retryInterval: 1m + timeout: 5m + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./apps/ + # add-highlight-start + images: + - name: ghcr.io/polinetworkorg/ + newTag: << inputs._image.tag | quote >> + digest: << inputs._image.digest | quote >> + # add-highlight-end ``` + +:::caution Input names +The inputs are named after the provider, with hyphens turned into underscores: +the provider `bot-ts-image` becomes `inputs.bot_ts_image`. ::: +In `deployment.yaml`, keep `image: ghcr.io/polinetworkorg/:latest`: the `name` +under `images` must match it exactly, and Flux replaces it with the pinned digest. +See `clusters/k3s/apps/web.yaml` for a complete example. + ## Options -These are some useful options you can configure by editing the `config.json` file. +These are some useful operations on existing apps. + ### Disable an app -If you want to disable an app (excluding it from the git generator) you can add `"disabled": true`. -Example: -```yaml title="config.json" -{ -# add-highlight-next-line - "disabled": true - "image": { - "image_list": "x/image1:latest x/image2:latest", - "update_strategy": "digest" - } -} +To stop an app temporarily, **suspend** its Kustomization in the +[Flux Web UI](https://flux.polinetwork.org) and scale it down from the node: + +```sh +kubectl scale deployment/ -n --replicas=0 +``` + +To bring it back, **resume** the Kustomization: Flux restores the replicas from Git. + +### Remove an app +1. Remove `.yaml` from `clusters/k3s/apps/kustomization.yaml` and delete the + file and the `apps//` folder. Remove its provider and receiver entry too, + if it had automatic image updates. +2. Once merged, Flux deletes the app's resources, except the ones annotated with + `prune: disabled`: the namespace, the PVCs and, for apps with automatic image + updates, the generated Kustomization. Delete those by hand from the node once you're + sure the data is not needed: + +```sh +kubectl delete kustomization -n flux-system # only for ResourceSet apps +kubectl delete namespace ``` -To reactivate the application just remove this property or set it to `false`. -This configuration option is introduced with [this commit](https://github.com/PoliNetworkOrg/terraform/commit/8ddf6e984c802d823573e85f147d48a916556c7c). +Volume directories stay on disk anyway (`reclaimPolicy: Retain`), under +`/srv//volumes//`. :::info -In the git generator configuration the selector is set to `"false"` due to the limitations of `matchExpression`. -However, in the application's `config.json` you can set the `disabled` flag to either `false` or `"false"`. +The `config.json` files with `"disabled": true` you may find at the root of polinetwork-cd +belong to the old ArgoCD setup. Flux doesn't read them. ::: From 2ee97deef1e34653d06dfbe17c0ebed873904966 Mon Sep 17 00:00:00 2001 From: Lorenzo Corallo Date: Sat, 10 Oct 2026 15:29:36 +0200 Subject: [PATCH 2/4] docs(infrastructure): address review comments - ci: test.yml runs on pushes to main and pull requests - setup: descriptive Azure CLI link text - secrets: correct what deletionPolicy Retain does - storage: ReadWriteOnce limits nodes, not pods - backups: create the age identity file with umask 077 --- docs/infrastructure/02-getting-started/pipelines/ci.md | 2 +- docs/infrastructure/02-getting-started/setup.md | 2 +- docs/infrastructure/03-Guides/add-a-new-secret.md | 2 +- docs/infrastructure/03-Guides/add-storage.md | 5 +++-- docs/infrastructure/03-Guides/backups.md | 5 +++-- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/infrastructure/02-getting-started/pipelines/ci.md b/docs/infrastructure/02-getting-started/pipelines/ci.md index 12d0786..c5c2432 100644 --- a/docs/infrastructure/02-getting-started/pipelines/ci.md +++ b/docs/infrastructure/02-getting-started/pipelines/ci.md @@ -7,7 +7,7 @@ title: CI The first step of our CI processes is the GitHub Actions workflow in each repository. For example, the [web](https://github.com/PoliNetworkOrg/web/tree/main/.github/workflows) repository has: -- `test.yml`, which typechecks and lints every push and pull request; +- `test.yml`, which typechecks and lints pushes to `main` and pull requests; - `docker.yml`, which on every push to `main` builds the Docker image for `amd64` and `arm64` and publishes it to GHCR as `ghcr.io/polinetworkorg/web:latest`. Our K3s node is ARM64, so the `arm64` build is required. Keep publishing a diff --git a/docs/infrastructure/02-getting-started/setup.md b/docs/infrastructure/02-getting-started/setup.md index b42115b..b4c49f8 100644 --- a/docs/infrastructure/02-getting-started/setup.md +++ b/docs/infrastructure/02-getting-started/setup.md @@ -73,7 +73,7 @@ resource group `rg-polinetwork`). Use them only for emergencies. You still need the Azure CLI to manage secrets in Key Vault and to inspect Azure resources. -- Download and install the Azure CLI [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). +- Download and install the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest). - Run ```az login``` to login. diff --git a/docs/infrastructure/03-Guides/add-a-new-secret.md b/docs/infrastructure/03-Guides/add-a-new-secret.md index 36de67c..0ad49bd 100644 --- a/docs/infrastructure/03-Guides/add-a-new-secret.md +++ b/docs/infrastructure/03-Guides/add-a-new-secret.md @@ -63,7 +63,7 @@ spec: target: name: azure-kv # name of the Kubernetes Secret ESO creates creationPolicy: Owner - deletionPolicy: Retain # keep the Secret if Key Vault is unreachable + deletionPolicy: Retain # keep the Secret if the Key Vault secret is deleted data: - secretKey: # key inside the Kubernetes Secret remoteRef: diff --git a/docs/infrastructure/03-Guides/add-storage.md b/docs/infrastructure/03-Guides/add-storage.md index 787e8a5..afdfdfc 100644 --- a/docs/infrastructure/03-Guides/add-storage.md +++ b/docs/infrastructure/03-Guides/add-storage.md @@ -22,8 +22,9 @@ There are two storage classes, one per data disk: grow until the whole disk is full, and then every app on that disk breaks. Check the free space first, and keep an eye on the `PersistentVolumeHigh` and `NodeDiskHigh` alerts. -- **Volumes are `ReadWriteOnce`** and live on one node: only one pod at a time should - write to them. +- **`ReadWriteOnce` doesn't mean one pod.** It limits a volume to one node, and we only + have one node: two pods can mount the same volume and write to it at the same time. + Make sure only one pod at a time uses it (see `Recreate` below). - **Volumes are retained.** Deleting a PVC doesn't delete its directory (`reclaimPolicy: Retain`). ::: diff --git a/docs/infrastructure/03-Guides/backups.md b/docs/infrastructure/03-Guides/backups.md index ea75fd4..22b1600 100644 --- a/docs/infrastructure/03-Guides/backups.md +++ b/docs/infrastructure/03-Guides/backups.md @@ -114,8 +114,9 @@ Copy the encrypted archive to your machine and decrypt it there with ```sh scp pnadmin@10.43.1.4:.tar.gz.age . -az keyvault secret show --vault-name kv-pn-infra --name backup-age-identity \ - --query value -o tsv > age-identity.txt +# umask 077: the identity file is readable only by you +(umask 077; az keyvault secret show --vault-name kv-pn-infra --name backup-age-identity \ + --query value -o tsv > age-identity.txt) mkdir restore age --decrypt --identity age-identity.txt .tar.gz.age | tar -xz -C restore From aa2ed513c226f7ea7ef1634ea3144b25c6db001f Mon Sep 17 00:00:00 2001 From: Lorenzo Corallo Date: Sat, 10 Oct 2026 15:30:26 +0200 Subject: [PATCH 3/4] docs(infrastructure): deprecate mounting secrets as files --- .../infrastructure/03-Guides/add-a-new-secret.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/infrastructure/03-Guides/add-a-new-secret.md b/docs/infrastructure/03-Guides/add-a-new-secret.md index 0ad49bd..9acc226 100644 --- a/docs/infrastructure/03-Guides/add-a-new-secret.md +++ b/docs/infrastructure/03-Guides/add-a-new-secret.md @@ -150,7 +150,14 @@ Append an entry under `data`: # add-highlight-end ``` -### As a file +### As a file (deprecated) + +:::warning Deprecated +Use [environment variables](#as-an-environment-variable) instead. Mounting the +Secret as a volume was required by the old CSI driver; with ESO it's only extra +YAML, and no app does it anymore. Keep it for an app that can only read its +configuration from files. +::: ```yaml title="apps//deployment.yaml" spec: @@ -171,9 +178,6 @@ Append an entry under `data`: # add-highlight-end ``` -Unlike the old CSI driver, you don't need to mount a volume to use a secret as -an environment variable: pick whichever form the app needs. - ## Deploy and check Open a PR into polinetwork-cd. Once it's merged, Flux applies it. Then, from the node: @@ -202,8 +206,8 @@ Environment variables are read only when a container starts, so restart the app: kubectl rollout restart deployment/ -n ``` -Secrets mounted as files are updated in place after a short delay, but your app -has to re-read them. +(Secrets mounted as files are updated in place after a short delay, but the app +has to re-read them.) ## Key Vault management From f4f04b1482568d86e74dc20f4d33855deda245a6 Mon Sep 17 00:00:00 2001 From: Lorenzo Corallo Date: Sat, 10 Oct 2026 15:31:16 +0200 Subject: [PATCH 4/4] feat: syntax highlighting for shell code blocks --- docusaurus.config.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docusaurus.config.ts b/docusaurus.config.ts index 39b87ac..ed55697 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -149,6 +149,8 @@ const config: Config = { prism: { theme: materialThemes.darker, darkTheme: materialThemes.darker, + // bash also registers the `sh` and `shell` aliases + additionalLanguages: ['bash'], magicComments: [ { className: 'theme-code-block-highlighted-line',