diff --git a/src/pages/docs/self-hosting/helm.mdx b/src/pages/docs/self-hosting/helm.mdx index 5453bce6a..0a4a250ce 100644 --- a/src/pages/docs/self-hosting/helm.mdx +++ b/src/pages/docs/self-hosting/helm.mdx @@ -34,8 +34,8 @@ What the chart supports today: | Level | Covers | |---|---| -| Supported: tested in CI, fixed in patch releases | Kubernetes 1.27 to 1.37 and kind; Helm 3.22 and 4.3; the datastore versions in [External datastores](#external-datastores); Gateway API, Ingress and LoadBalancer exposure; bundled datastores for evaluation | -| Community: expected to work, not covered by CI | GKE, EKS and AKS (`examples/cloud/`), OpenShift, k3s, Docker Desktop and OrbStack; Helm 3 from 3.10; Argo CD and Flux; PgBouncer and a read replica for PostgreSQL; managed Redis; Google Cloud Storage through HMAC keys | +| Supported: tested in CI, fixed in patch releases | Kubernetes 1.27 to 1.37 and kind; Helm 3.22 and 4.3; Argo CD 3.6.1 and Flux 2.10.0 ([bounded controller acceptance](#gitops-upgrades)); the datastore versions in [External datastores](#external-datastores); Gateway API, Ingress and LoadBalancer exposure; bundled datastores for evaluation | +| Community: expected to work, not covered by CI | GKE, EKS and AKS (`examples/cloud/`), OpenShift, k3s, Docker Desktop and OrbStack; Helm 3 from 3.10; other Argo CD and Flux versions; PgBouncer and a read replica for PostgreSQL; managed Redis; Google Cloud Storage through HMAC keys | | Not supported today | The code sandbox in a `baseline` or `restricted` Pod Security namespace (it is privileged); TLS-only ClickHouse endpoints such as ClickHouse Cloud, and replicated ClickHouse clusters; Temporal Cloud; IAM roles for service accounts (IRSA), Workload Identity and Entra ID; Azure Blob Storage; TLS Redis for the LLM gateway and fi-collector; IAM database authentication; bundled datastores in production | The full matrix is in the chart README under [Supported configurations](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#supported-configurations). @@ -50,20 +50,24 @@ Data from a Docker Compose install, Standalone or Distributed, cannot move to He ## Verify the chart -The release workflow signs each published chart keylessly with cosign (GitHub OIDC, no key to manage) and attaches a build provenance attestation. Both name the workflow and the tag that built the chart. Check them with `cosign` and the GitHub CLI before you install: +The release workflow signs each published chart keylessly with cosign (GitHub OIDC, no key to manage) and attaches a build provenance attestation. Both name the workflow and the tag that built the chart. Resolve the exact version once with ORAS, verify its digest with `cosign` and the GitHub CLI, and deploy that same digest rather than resolving the tag again: ```bash VERSION=X.Y.Z -cosign verify --new-bundle-format=false ghcr.io/future-agi/charts/futureagi:$VERSION \ +REPOSITORY=ghcr.io/future-agi/charts/futureagi +DIGEST=$(oras resolve "$REPOSITORY:$VERSION") +cosign verify --new-bundle-format=false "$REPOSITORY@$DIGEST" \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ - --certificate-identity https://github.com/future-agi/future-agi/.github/workflows/helm-release.yml@refs/tags/v$VERSION -gh attestation verify oci://ghcr.io/future-agi/charts/futureagi:$VERSION --repo future-agi/future-agi \ + --certificate-identity "https://github.com/future-agi/future-agi/.github/workflows/helm-release.yml@refs/tags/v$VERSION" +gh attestation verify "oci://$REPOSITORY@$DIGEST" --repo future-agi/future-agi \ --signer-workflow future-agi/future-agi/.github/workflows/helm-release.yml ``` +Stop if any verification fails. Authentication establishes publisher identity and artifact integrity; it does not establish database migration compatibility or backup readiness. Helm 4 can pull and install by digest. For Helm 3, use a verified local chart archive rather than returning to a mutable tag. + `--new-bundle-format=false` makes cosign check the chart's signature, as the release workflow does. Without it, cosign checks the provenance attestation in its place, which `gh attestation verify` already covers. -The chart pins each Future AGI image by digest while it runs the chart's own release tag (`image.digests`, the digests the release built), so a verified chart also fixes the exact images it runs. The images themselves are not signed yet: see [Verifying an image](/docs/self-hosting/images#verifying-an-image). +A published chart pins its default Future AGI images by digest (`image.digests`, the digests the release built). Review your overrides: changing `image.tag`, a component tag/repository/digest, or `image.pinDigests` can change the images selected. A mirror must preserve the authenticated digests. The images themselves are not signed yet: see [Verifying an image](/docs/self-hosting/images#verifying-an-image). The GitHub Release `vX.Y.Z` carries the same package (`futureagi-X.Y.Z.tgz`), its signature bundle, checksums (`futureagi-X.Y.Z.sha256`) and the list of images the chart can pull, for registries that cannot proxy GHCR: `futureagi-images-X.Y.Z.txt`, one `repository:tag@sha256:...` per line, and a Hauler manifest, `futureagi-hauler-X.Y.Z.yaml`, for mirroring them into a private registry. The chart README's [Verify](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#verify) section shows how to check that copy, and its [GitOps](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#gitops-argo-cd-and-flux) section shows how Flux checks the signature before every install and upgrade. @@ -283,7 +287,7 @@ Set everything in `my-values.yaml` and apply it with `helm upgrade`, passing the - **Enterprise.** `edition=ee` with `license.existingSecret` (a Secret holding `EE_LICENSE_KEY`) turns on the Enterprise features in the same public image. `examples/enterprise.yaml` adds SSO (`auth.google`, `auth.github`, `auth.microsoft`), a corporate proxy (`global.proxy`; the LLM gateway does not use it yet), a CA bundle (`global.caBundle`) and a private registry mirror (`global.imageRegistry`). See [Enterprise](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#enterprise). - **Private provider URLs.** `agentccGateway.allowPrivateProviderURLs=true` lets organization providers use base URLs on private networks, such as an Ollama or vLLM server in the cluster or on your network. It sets `AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS` on the backend and the gateway. Loopback, link-local and cloud metadata addresses stay refused. Set it only when everyone who can add a provider is trusted: it puts every in-cluster service in reach of a provider URL. See [Private and local provider URLs](/docs/command-center/features/self-hosted-models#private-and-local-provider-urls). - **Gateway shutdown.** A stopping gateway pod gets 45 s (`agentccGateway.terminationGracePeriodSeconds`): the 10 s preStop sleep (Kubernetes 1.30 or newer), 30 s of `agentccGateway.config.server.shutdown_timeout` for in-flight streams, and 5 s in which the gateway sends its buffered request logs. If you change either setting, keep the grace period at least their sum plus 5 s, and add up to 15 s with `agentccGateway.config.otel.exporter: otlp`. -- **GitOps.** Argo CD renders the chart with `helm template`, so it needs `secrets.existingSecret` and explicit passwords for bundled datastores. Flux can verify the chart's signature before every install and upgrade. Pin an exact chart version in either. `examples/gitops/` has an Argo CD Application and a Flux HelmRelease. See [GitOps](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#gitops-argo-cd-and-flux). +- **GitOps.** Use exactly one deployment owner: direct Helm, Argo CD or Flux. Pin the verified chart digest, keep credentials in existing Secrets, and promote changes through a protected Git review. Argo CD needs external signature verification before promotion; Flux can enforce the exact cosign identity in its source controller. See [GitOps upgrades](#gitops-upgrades) below. - **Anything else.** `config.extraEnv` sets environment variables on the backend, workers and bootstrap job. The [Configuration reference](/docs/self-hosting/configuration/reference) lists every supported key; those marked H apply to the chart. To add an LLM key to a running install within the same chart version, reuse the release's values: @@ -297,29 +301,56 @@ This restarts the application pods, not the bundled datastores. `--reset-then-re ## Upgrade and roll back -Chart `X.Y.Z` runs images `vX.Y.Z`, so upgrading the chart upgrades the platform. Any patch within a minor is a supported path, and so is the previous minor to the next. Going further, upgrade one minor at a time, and read the [release notes](https://github.com/future-agi/future-agi/releases) first: they carry a migrations callout when a release has one. +Chart `X.Y.Z` runs images `vX.Y.Z`, so upgrading the chart upgrades the platform. Before every transition, review the [release notes](https://github.com/future-agi/future-agi/releases), migrations and value changes, and identify a tested, coordinated recovery point for your datastores and application keys. A patch or minor version number is not proof that old and new code can share a schema. + +The old application pods can keep writing while bootstrap runs. If their compatibility with the migration cannot be established, plan a maintenance outage rather than assuming hook ordering quiesces writers. Keep one deployment owner; do not run direct Helm commands against an Argo- or Flux-managed release. + +For a direct Helm release, [verify the target chart](#verify-the-chart) and use the resulting `REPOSITORY` and `DIGEST`. With Helm 4: ```bash -NEW=X.Y.Z -helm upgrade futureagi oci://ghcr.io/future-agi/charts/futureagi --version "$NEW" \ - -n futureagi -f medium.yaml -f my-values.yaml --timeout 20m --rollback-on-failure # Helm 3: --atomic +helm status futureagi --kube-context CONTEXT -n futureagi +helm history futureagi --kube-context CONTEXT -n futureagi +helm upgrade futureagi "oci://$REPOSITORY@$DIGEST" \ + --kube-context CONTEXT -n futureagi -f medium.yaml -f my-values.yaml --wait --timeout 20m +helm test futureagi --kube-context CONTEXT -n futureagi --logs --timeout 5m ``` -Pass every values file you installed with, in the same order: here the preset and `my-values.yaml`, or `bundled.yaml` for an evaluation install. A file you leave out takes its settings with it. Never use `--reuse-values` across versions: it keeps the previous release's merged values, so new keys and changed defaults in the new chart are silently dropped. +Pass every values file you installed with, in the same order: here the preset and `my-values.yaml`, or `bundled.yaml` for an evaluation. Never use `--reuse-values` across versions. Do not enable `--rollback-on-failure` (Helm 4) or `--atomic` (Helm 3) unless the exact transition is known to tolerate an image-only rollback. -The bootstrap job runs first, with the new image: migrations, seeds, the ClickHouse schema, CDC and schedules. Only then do the Deployments roll. A failed bootstrap stops the upgrade before any Deployment rolls, and its log says why: +A reviewed source checkout containing `deploy/helm/futureagi/hack/prepare-release.py` offers the same authentication boundary as a single preparation step. It requires Python 3.9+, PyYAML, Helm 4, ORAS and cosign, and writes a verified local chart and `release-lock.json` into a new private directory. It does not apply anything to the cluster, qualify migrations or authorize data restoration: ```bash -kubectl -n futureagi logs job/futureagi-bootstrap +python3 deploy/helm/futureagi/hack/prepare-release.py \ + --version X.Y.Z --destination "$HOME/futureagi-release-X.Y.Z" ``` -Fix the cause and run the upgrade again, or go back with `helm rollback futureagi -n futureagi`, which restores the previous revision's Deployments and Secret. A job still running after Helm's `--timeout` must finish before you retry. +With that output, replace the OCI reference in `helm upgrade` with `$HOME/futureagi-release-X.Y.Z/futureagi-X.Y.Z.tgz`. Use the [operator runbook](https://github.com/future-agi/future-agi/blob/dev/deploy/helm/futureagi/UPGRADING.md) from the same reviewed source revision. + +The upgrade bootstrap runs the target image's migrations, seeds, ClickHouse schema, CDC and schedules before the new Deployments roll. A failed bootstrap can still leave changed data. Inspect the Job and its logs, and wait for a terminal state before retrying or recovering; a Helm timeout does not stop the Job: + +```bash +kubectl --context CONTEXT -n futureagi get job futureagi-bootstrap +kubectl --context CONTEXT -n futureagi logs job/futureagi-bootstrap +``` -`helm rollback` restores images and the Secret, never the database schema. A rollback across a migration the old version cannot run on means restoring the pre-upgrade backup, so take one before every minor upgrade. +`helm rollback` does not restore the database schema or externally managed Secrets. Use image/configuration rollback only after checking that the previous application can run against the current data. Otherwise stop writers and use your deployment-specific data restore procedure. There is no universal Kubernetes restore command or zero-loss promise. -The chart README's [Upgrading](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#upgrading) section covers renamed values and breaking changes. +### GitOps upgrades + +Use the reviewed [GitOps examples](https://github.com/future-agi/future-agi/tree/dev/deploy/helm/futureagi/examples/gitops) and promote the prepared digest and signing identity together. Do not combine this with a floating semver range or another image updater. The Compose automatic updater's compatibility proofs do not authorize Kubernetes updates. + +- **Argo CD:** use a version with native OCI source support, with `repoURL: oci://ghcr.io/future-agi/charts/futureagi`, `path: .` and the verified digest as `targetRevision`. Argo does not enforce this cosign policy: verification before protected Git promotion is the trust boundary. The operator creates the namespace and scoped AppProject. Pre-create the application keys and datastore credentials; `helm template` cannot preserve generated secrets with `lookup`. The example disables pruning, self-heal and failure retries. Never use selective sync or `ApplyOutOfSyncOnly`, which can skip bootstrap hooks. +- **Flux:** set `OCIRepository.spec.ref.digest` and an exact release-tag OIDC subject, rather than a broad `v.*` signer match. Apply the namespace-scoped reconciler Role/ServiceAccount first; the application receives no cluster-administration token. The example enables Helm tests and selects `RemediateOnFailure` with zero retries and no last-failure remediation for install and upgrade. Failed migration is not an instruction to uninstall or automatically roll back. + +**Already using Flux?** These examples describe a new namespace-scoped installation, not a migration of release ownership or storage. Preserve the existing HelmRelease name/namespace, release name, target namespace and storage namespace unless you have a separately reviewed migration plan. Do not delete/recreate the old HelmRelease or add a second one to adopt the example: that can uninstall the old release or leave two reconcilers managing it. Moving out of a control namespace such as `flux-system` also needs a review of Secret storage and ServiceAccount/RoleBinding scope. + +For external stores, Argo runs bootstrap in PreSync. Bundled stores are in Sync wave -2, bootstrap in wave -1 and application workloads in wave 0. These waves do not stop existing writers. Status must show the desired digest, the current observed digest/generation and the current operation or test result—not merely an old green Ready condition. + +On failure, disable automated sync or suspend reconciliation **in Git**, then confirm the in-flight operation and bootstrap Job have stopped. Preserve diagnostics without publishing Secret values. Choose forward repair, reviewed image/configuration rollback, or coordinated data restore based on the actual state. Set Git to the recovered release before resuming, then verify login, existing traces/evaluations and a retained file. + +The controller acceptance suite uses Argo CD 3.6.1 and Flux 2.10.0 on disposable kind clusters, four source-built product images and synthetic store/key/PVC witnesses. Its same-source chart fixtures and test signing keys prove controller mechanics, not arbitrary cross-version database migration rollback, external-store recovery or optional serving/code-executor behavior. ## Back up and restore @@ -338,15 +369,14 @@ Without `INTEGRATION_ENCRYPTION_KEY`, the integration credentials in a restored kubectl -n futureagi get secret futureagi-secrets -o yaml > futureagi-secrets.backup.yaml ``` -To restore: +Plan restoration for your actual datastore operators, snapshots and storage layout: -1. Scale the application down so nothing writes meanwhile: `kubectl -n futureagi scale deploy -l app.kubernetes.io/instance=futureagi --replicas=0`. -2. Restore PostgreSQL, then ClickHouse from a backup taken at or after the same point, then the bucket. -3. Put back the application Secret. -4. Run `helm upgrade` with the same chart version and the same values files, in the same order. The bootstrap job re-applies the schema, CDC and Temporal schedules, and the Deployments come back. Autoscaled Deployments stay at 0, so scale them to 1 yourself. -5. Run `helm test futureagi -n futureagi`, sign in, open a recent trace and download a file. +1. Pause the deployment owner in its source of truth and wait for active operations and bootstrap Jobs to finish. Stop all application writers and prevent reconciliation or autoscaling from recreating them during recovery. +2. Restore PostgreSQL, ClickHouse, object storage, Temporal persistence and application keys to the coordinated recovery point your restore procedure establishes. A later ClickHouse backup is not automatically consistent with an earlier PostgreSQL backup. +3. Choose the application/chart revision that is compatible with the restored data. Preserve the original encryption key and any externally managed Secrets. Make that revision and the complete values files the deployment owner's desired state before resuming it. +4. Check bootstrap and rollout outcomes, then sign in, open a retained trace/evaluation and download a retained file. With direct Helm or Flux, check Helm test results too; Argo does not run those Helm test hooks. -For bundled datastores, Velero or your volume snapshots cover the PersistentVolumeClaims. Stop the application first for a consistent copy. See [Backup and restore](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#backup-and-restore) in the chart README. +For bundled datastores, use a tested PVC backup/restore process such as Velero or storage snapshots. Snapshot creation alone does not prove application-consistent recovery. See [Backup and restore](https://github.com/future-agi/future-agi/blob/main/deploy/helm/futureagi/README.md#backup-and-restore) in the chart README. ## Troubleshooting