Skip to content
Merged
9 changes: 9 additions & 0 deletions app/en/operate/deploy/helm/_meta.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
import type { MetaRecord } from "nextra";

export const meta: MetaRecord = {
"internal-networking": {
title: "Internal networking",
},
"ipv6-dual-stack": {
title: "IPv6 and dual-stack",
},
"container-registry": {
title: "Use your own container registry",
},
"values-reference": {
title: "Values reference",
},
Expand Down
133 changes: 133 additions & 0 deletions app/en/operate/deploy/helm/container-registry/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
---
title: "Use your own container registry"
description: "Pull the Helm-installed Arcade platform's images from your own registry mirror, including for air-gapped installs"
---

import { Callout } from "nextra/components";

# Use your own container registry

This guide covers pulling every image the [Helm-installed Arcade platform](/operate/deploy/helm) needs from a registry you run, instead of Docker Hub.

## When you need it

- Your cluster can't reach Docker Hub, or your policy requires images from an approved registry.
- You run an air-gapped install.
- You mirror images to avoid Docker Hub rate limits.

## Values overlay

There's no single registry prefix for the whole chart: set each image's repository to its path in your registry. Copy the images at the tags your chart version uses (see the values reference on [Artifact Hub](https://artifacthub.io/packages/helm/arcade/arcade)).

```yaml
dockerRegistry:
enabled: true
existingSecret: arcade-registry-pull # a kubernetes.io/dockerconfigjson Secret

engine:
image:
repository: registry.example.com/arcadedev/engine
coordinator:
image:
repository: registry.example.com/arcadedev/coordinator
dashboard:
image:
repository: registry.example.com/arcadedev/dashboard
experienceApi:
image:
repository: registry.example.com/arcadedev/experience-api
workerDefaults:
image:
repository: registry.example.com/arcadedev/worker
rbac:
openfga:
image:
repository: registry.example.com/arcadedev/openfga
postgres:
image: registry.example.com/postgres:17-alpine

supportImages:
busybox:
repository: registry.example.com/busybox
curl:
repository: registry.example.com/curlimages/curl
postgres:
repository: registry.example.com/postgres
tag: 17-alpine

# Only if Arcade Deploy is enabled
deployments:
image:
repository: arcadedev/runner
registry:
server: registry.example.com
imagePullSecrets: [arcade-registry-pull]
```

### Pull credentials

`dockerRegistry` controls the pull Secret on the Arcade pods:

- `dockerRegistry.enabled` (default `true`) adds an `imagePullSecrets` entry to the Arcade component pods (Arcade Engine, coordinator, dashboard, Experience API, OpenFGA, workers, and the chart's Jobs). Leave it on for the private `arcadedev/*` images or a registry that needs credentials.
- `dockerRegistry.existingSecret` names a Secret you create in the release namespace, of type `kubernetes.io/dockerconfigjson`. This is the usual choice for a mirror.
- Without `existingSecret`, the chart generates a Secret named `dockerRegistry.secretName` from `dockerRegistry.server`, `dockerRegistry.username`, and `dockerRegistry.token` (a Docker Hub access token).

<Callout type="warning">
The bundled PostgreSQL and Redis StatefulSets don't get a pull Secret. If your mirror requires authentication, they can't pull from it. Mirror `postgres.image` to a path your nodes can pull without credentials (for example, through node-level registry credentials), or use external PostgreSQL.

The bundled Redis image is fixed at `docker.io/library/redis:8-alpine`; the chart has no value to change it. To avoid pulling it from Docker Hub, configure a node-level mirror for `docker.io` in your container runtime, or use external Redis (`redis.enabled: false`).
</Callout>

### Support images

The init containers use `supportImages.busybox` (`generate-env-file`), `supportImages.curl` (`wait-for-workers`, which must be a real curl image), and `supportImages.postgres` (`wait-for-db`, which needs the `psql` client). Each has `repository`, `tag`, and `pullPolicy`. An empty `supportImages.postgres.repository` falls back to `postgres.image`, so a mirrored `postgres.image` already covers it.

### The Arcade Deploy runner image

If you enable [Arcade Deploy](/build/arcade-deploy), the Arcade Engine starts every deployed server from the runner image in the workers namespace:

- The image is `deployments.registry.server` joined with `deployments.image.repository` (default `arcadedev/runner`). With `deployments.registry.server` empty, the repository is used as-is. Its tag is `deployments.image.tag`, or the chart's app version when that's empty.
- Deployed server pods use the Secrets listed in `deployments.registry.imagePullSecrets`. These Secrets must exist in the workers namespace (`deployments.namespace`), not the release namespace.
- When `dockerRegistry.enabled` is on and `dockerRegistry.existingSecret` is empty, the chart also adds `dockerRegistry.secretName` to that list. A Secret you name in `dockerRegistry.existingSecret` isn't added, so list it in `deployments.registry.imagePullSecrets` and create a copy in the workers namespace.

## Air-gapped installs

1. Copy every image above, at your chart version's tags, into your registry. Include the Arcade Deploy runner image if you use Arcade Deploy, and any images your workers list overrides.
2. Copy the chart itself (`oci://public.ecr.aws/s5i6x9d1/charts/arcade`) into an OCI registry you can reach, and install from there.
3. Create the pull Secret in the release namespace, and in the workers namespace for Arcade Deploy.
4. Install with the overlay above.

Image pulls are only part of an air-gapped install: the identity provider, OAuth providers, and any MCP server dependencies also need to be reachable from inside your network.

## Verify it

```bash
# Every image comes from your registry
kubectl -n arcade get pods -o jsonpath='{range .items[*]}{range .spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

# Pods pulled successfully
kubectl -n arcade get pods
kubectl -n arcade get events --field-selector reason=Failed
```

For Arcade Deploy, deploy a test server and run the same image check in the workers namespace (`arcade-workers` by default).

## Common failures

| Symptom | Likely cause and fix |
| --- | --- |
| `ImagePullBackOff` on one component | Its `image.repository` still points at Docker Hub, or the tag isn't mirrored. Check the pod's image and mirror the missing tag. |
| `ImagePullBackOff` on init containers | `supportImages.*` still point at Docker Hub. Override them. |
| `401` or `unauthorized` pulling PostgreSQL or Redis | The bundled StatefulSets have no pull Secret, and Redis always pulls from Docker Hub. See the warning under [Pull credentials](#pull-credentials). |
| `401` or `unauthorized` pulling | The pull Secret is missing, in the wrong namespace, or not `kubernetes.io/dockerconfigjson`. Check `dockerRegistry.existingSecret`. |
| Deployed servers fail to pull the runner image | The Secret isn't in the workers namespace, or isn't listed in `deployments.registry.imagePullSecrets`. |
| The runner image path is doubled, for example `registry.example.com/registry.example.com/...` | `deployments.image.repository` includes the registry host while `deployments.registry.server` is also set. Set the host in one place. |

## What you own and what the chart handles

| You | The chart |
| --- | --- |
| Running the registry, and mirroring each image at the right tag | Using the repository and tag you set for every image |
| Creating pull Secrets in the release namespace and the workers namespace | Adding `imagePullSecrets` to Arcade pods when `dockerRegistry.enabled` |
| Mirroring the chart for air-gapped installs | Generating a pull Secret from `dockerRegistry.username` and `dockerRegistry.token` when you don't bring one |
| Keeping mirrors current on each upgrade | Building the runner image path from `deployments.registry.server` and passing pull Secrets to deployed servers |
112 changes: 112 additions & 0 deletions app/en/operate/deploy/helm/internal-networking/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
title: "Internal networking"
description: "Run a Helm-installed Arcade platform with no public ingress, behind an internal load balancer or Gateway API, and let it reach MCP servers on your internal network"
---

import { Callout } from "nextra/components";

# Internal networking

This guide covers running the [Helm-installed Arcade platform](/operate/deploy/helm) entirely inside your network: no public ingress, traffic through an internal load balancer or a Gateway API implementation you run, and MCP servers on internal addresses that the Arcade Engine is allowed to reach.

## When you need it

- Arcade must not be reachable from the internet, only from your corporate network or VPN.
- Your cluster already exposes internal services through a Gateway API implementation or an internal ingress controller.
- MCP servers or workers that Arcade calls live on private addresses (for example `10.0.0.0/8` or `*.corp.internal`).

## Values overlay

The chart doesn't create a load balancer itself. Every Arcade Service is `ClusterIP` by default (`engine.service.type`, `coordinator.service.type`, `dashboard.service.type`, `experienceApi.service.type`), and traffic enters through routes the chart renders for your own Gateway or ingress controller. Making that entry point internal is a property of the Gateway or ingress controller you attach to.

### Gateway API (recommended)

The chart renders `HTTPRoute`s, never the `Gateway`; you run that. Whenever `gateway.enabled` is `true` and parent refs are set, the chart also renders an AWS Load Balancer Controller `TargetGroupConfiguration` (`gateway.k8s.aws/v1beta1`) per service to set its health check path. It doesn't check whether that CRD is installed, so on a cluster without the AWS Load Balancer Controller (GKE, AKS, on-premises), `helm install` and `helm upgrade` fail with `no matches for kind TargetGroupConfiguration`. Charts up to 1.11.1 behave this way; a future chart release renders it only when the cluster serves the CRD. Point the routes at an internal Gateway, one whose listener is backed by an internal load balancer in your environment:

```yaml
gateway:
enabled: true
parentRefs:
- name: arcade-internal-gw
namespace: gateway-system
sectionName: https
hostname: arcade.corp.example.com
protocol: https

ingress:
enabled: false
```

A service's route renders only when it has parentRefs: its own `gateway.<service>.parentRefs` if set, otherwise the shared `gateway.parentRefs`. With both empty, that service gets no route. Services get hostnames from `gateway.subdomains.*` (`api`, `coordinator`, `dashboard`, `experienceApi`) under `gateway.hostname`. To attach one service to a different Gateway, set its own `gateway.<service>.parentRefs` (`engine`, `coordinator`, `dashboard`, `experienceApi`), which replaces the shared list for that service. A service with its own `parentRefs` gets a route even when the shared list is empty. If your controller only serves `v1beta1` routes, set `gateway.apiVersion`.

### Legacy Ingress

The `ingress.*` path is deprecated in favor of `gateway.*`, but still works. Use the private ingress class, the one backed by your internal load balancer:

```yaml
ingress:
enabled: true
type: private
privateIngressClass: nginx-internal
hostname: arcade.corp.example.com
privateAnnotations:
cert-manager.io/cluster-issuer: selfsigned-ca-issuer
nginx.ingress.kubernetes.io/ssl-redirect: "true"
```

`ingress.type: private` (the default) selects `ingress.privateIngressClass` and `ingress.privateAnnotations`. With `ingress.selfSignedIssuer.enabled` (on by default), the chart also creates a self-signed ClusterIssuer named by `ingress.selfSignedIssuer.name`; replace it with your internal CA's issuer if you have one.

### Let the Arcade Engine reach internal MCP servers

By default the Arcade Engine refuses to dial private and internal addresses from user-created worker and MCP server URLs. List the destinations it may reach in `engine.ssrfAllowlist`:

```yaml
engine:
ssrfAllowlist:
- https://tools.corp.example.com:8443 # exact scheme + host [+ port]
- "*.corp.internal" # any subdomain of the suffix
- 10.20.0.0/16 # CIDR or bare IP, matched against the dialed IP
```

An empty list (the default) keeps the private-address block. `"*"` turns the check off for every destination; avoid it outside evaluation. Two related lists use the same entry shapes: `engine.authSsrfAllowlist` for tenant-created OAuth provider endpoints (empty by default), and `engine.pluginSsrfAllowlist` for plugin webhook hooks.

<Callout type="warning">
`engine.pluginSsrfAllowlist` defaults to `["*"]`, so plugin hooks can reach any internal address, cloud metadata included. On an internal network, narrow it to the hook endpoints you use, or set it to `[]` if no hook needs an internal address.
</Callout>

### Trust your internal proxy's forwarded headers

If TLS ends at your load balancer or Gateway, the coordinator needs to trust its `X-Forwarded-*` headers to build `https://` sign-in redirects. Set `coordinator.proxyHeaders.forwardedAllowIps` to the pod or Service CIDR of your ingress controller or Gateway data plane (comma-separated); the default trusts only `127.0.0.1`.

## Verify it

```bash
# Routes attached to your Gateway, and accepted by it
kubectl -n arcade get httproute
kubectl -n arcade describe httproute

# No public ingress or LoadBalancer Services from the chart
kubectl -n arcade get ingress,svc
```

Then, from a machine inside your network, open the dashboard hostname and sign in. From outside the network, the hostname shouldn't resolve or connect.

## Common failures

| Symptom | Likely cause and fix |
| --- | --- |
| No `HTTPRoute` for a service | `gateway.enabled` is `false`, or neither `gateway.parentRefs` nor that service's `gateway.<service>.parentRefs` is set. |
| Routes exist but don't serve traffic | The Gateway doesn't accept them. Check the route's `status.parents` conditions, the listener's `sectionName`, and whether the Gateway allows routes from the release namespace. |
| Sign-in redirects to `http://` or loops | TLS ends upstream and the coordinator doesn't trust the proxy. Set `coordinator.proxyHeaders.forwardedAllowIps`. |
| A worker or MCP server on a private address fails to connect | The Arcade Engine's private-address block refused it. Add the host, wildcard, or CIDR to `engine.ssrfAllowlist`. For an OAuth provider endpoint, use `engine.authSsrfAllowlist`. |
| Connections to an allowed internal address still time out with `networkPolicy.enabled: true` | The chart's egress policy for Arcade pods doesn't include that range. Add it to `networkPolicy.egressCIDRs`. |

## What you own and what the chart handles

| You | The chart |
| --- | --- |
| The `Gateway` or ingress controller, and the internal load balancer behind it | `HTTPRoute`s (or Ingress resources) for each Arcade service |
| Internal DNS records for the hostnames | `ClusterIP` Services for every component |
| TLS certificates or the issuer for your internal CA | A self-signed ClusterIssuer when `ingress.selfSignedIssuer.enabled` |
| Firewalls and network reachability between your network and the cluster | The Arcade Engine's private-address block and the allowlists you configure |
| Choosing which internal destinations Arcade may reach | Optional NetworkPolicies for Arcade pods (`networkPolicy.*`) |
Loading
Loading