From 152dcea688542030e02c1ae8a2918e53d28d966f Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:16 +0200 Subject: [PATCH 01/16] Target nebari-operator v0.1.1 and also test alpha.20 in CI (#40) The template developed, tested and documented against v0.1.0-alpha.19, while NIC v0.14.0 deploys v0.1.0-alpha.20 and the latest release is v0.1.1. - dev/Makefile: default OPERATOR_REF to v0.1.1 (overridable with ?=) and clone operator scripts into a ref-specific cache directory, so changing the ref never reuses scripts cloned for another version - test-integration.yaml: run the job for v0.1.1 and v0.1.0-alpha.20 - CRD reference: link the v0.1.1 types and state which release NIC deploys - README: describe the two CI operator versions --- .github/workflows/test-integration.yaml | 9 ++++++++- README.md | 4 +++- dev/Makefile | 5 +++-- docs/nebariapp-crd-reference.md | 5 +++-- docs/site/src/content/docs/nebariapp-crd-reference.md | 5 +++-- 5 files changed, 20 insertions(+), 8 deletions(-) diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index 1871387..79e6c55 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -14,9 +14,16 @@ jobs: nebariapp-integration: runs-on: ubuntu-latest timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + # v0.1.1 is the release the template documents; v0.1.0-alpha.20 is what + # NIC v0.14.0 deploys, so packs are also proven on today's clusters. + operator-ref: [v0.1.1, v0.1.0-alpha.20] + name: nebariapp-integration (${{ matrix.operator-ref }}) env: CLUSTER_NAME: pack-integration - OPERATOR_REF: v0.1.0-alpha.19 + OPERATOR_REF: ${{ matrix.operator-ref }} steps: - name: Checkout software-pack-template uses: actions/checkout@v5 diff --git a/README.md b/README.md index 8c78722..b450a20 100644 --- a/README.md +++ b/README.md @@ -624,7 +624,9 @@ file. Tests each example with `nebariapp.enabled=true` on a full Nebari infrastructure stack: - Creates a kind cluster with MetalLB, Envoy Gateway, cert-manager, and Keycloak -- Installs the nebari-operator from a pinned release (currently `v0.1.0-alpha.19`) +- Runs once per operator version: `v0.1.1` (the release these docs track) and + `v0.1.0-alpha.20` (what NIC v0.14.0 deploys) +- Configures the operator the way NIC does (`dev/configure-operator.sh`) - Deploys each example with NebariApp enabled and a `*.nebari.local` hostname - Verifies NebariApp reaches `Ready` condition (HTTPRoute created, TLS configured) - For auth-enabled examples (kustomize production, auth-fastapi), verifies diff --git a/dev/Makefile b/dev/Makefile index 9a143b3..05d763c 100644 --- a/dev/Makefile +++ b/dev/Makefile @@ -23,7 +23,8 @@ CHART_PODINFO := ../examples/wrap-existing-chart/chart CLUSTER_NAME := my-pack-dev HOSTNAME := my-pack.nebari.local CACHE_DIR := .cache -OPERATOR_REPO := $(CACHE_DIR)/nebari-operator +# Ref-specific so changing OPERATOR_REF never reuses scripts cloned for another version. +OPERATOR_REPO = $(CACHE_DIR)/nebari-operator-$(OPERATOR_REF) # MetalLB IP pool configuration template. # METALLB_BASE_IP is substituted at runtime with the Kind network base. @@ -68,7 +69,7 @@ _cluster-create: _clone-operator _kind-create _metallb _services _keycloak-setup @echo "" @echo "Run 'make up-basic' (or any up-* target) to deploy an example." -OPERATOR_REF := v0.1.0-alpha.19 +OPERATOR_REF ?= v0.1.1 _clone-operator: @if [ ! -d "$(OPERATOR_REPO)" ]; then \ diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 33f7c76..32b7a5b 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -4,8 +4,9 @@ Complete field-by-field reference for the NebariApp custom resource. **API Version:** `reconcilers.nebari.dev/v1` **Kind:** `NebariApp` -**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.0-alpha.19/api/v1/nebariapp_types.go) -**Operator version this doc tracks:** `v0.1.0-alpha.19` +**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/api/v1/nebariapp_types.go) +**Operator version this doc tracks:** `v0.1.1`. NIC v0.14.0 deploys `v0.1.0-alpha.20`; fields +added after that release are marked *(v0.1.0+)*. ## Full Example diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 2b01f54..9d569b4 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -6,8 +6,9 @@ Complete field-by-field reference for the NebariApp custom resource. **API Version:** `reconcilers.nebari.dev/v1` **Kind:** `NebariApp` -**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.0-alpha.19/api/v1/nebariapp_types.go) -**Operator version this doc tracks:** `v0.1.0-alpha.19` +**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/api/v1/nebariapp_types.go) +**Operator version this doc tracks:** `v0.1.1`. NIC v0.14.0 deploys `v0.1.0-alpha.20`; fields +added after that release are marked *(v0.1.0+)*. ## Full Example From b657132d4851666058b60135330a830c68ef1176 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:16 +0200 Subject: [PATCH 02/16] Configure the dev cluster like NIC so local auth works (#42) The Auth Flow page told developers to disable NebariApp locally, while the Makefile claimed the full stack works. Neither was right: the release install.yaml leaves the operator's in-cluster issuer at port 8080 with no /auth path, so Envoy Gateway rejected every SecurityPolicy and auth-enabled apps returned 500. The browser was also sent to in-cluster Keycloak URLs it cannot reach, and no per-app certificates were issued. - dev/configure-operator.sh sets KEYCLOAK_ISSUER_SERVICE_PORT, KEYCLOAK_ISSUER_CONTEXT_PATH, KEYCLOAK_EXTERNAL_URL and TLS_CLUSTER_ISSUER_NAME, as NIC does, and applies dev/keycloak-route.yaml to expose Keycloak at keycloak.nebari.local (with a Forwarded header so Keycloak builds https login URLs) - Makefile: run it during cluster setup, build chart dependencies before installing, add a keycloak hosts entry and a login-test target - dev/login-test.sh drives the OIDC login with curl and checks the app - CI configures the operator the same way - Docs: replace the local-development limitation, list up-podinfo and login-test, and add a troubleshooting entry for the 500 caused by a rejected SecurityPolicy --- .github/workflows/test-integration.yaml | 5 ++- README.md | 33 +++++++++++++++++--- dev/Makefile | 23 +++++++++++--- dev/configure-operator.sh | 25 +++++++++++++++ dev/keycloak-route.yaml | 31 +++++++++++++++++++ dev/login-test.sh | 41 +++++++++++++++++++++++++ docs/auth-flow.md | 7 +++-- docs/site/src/content/docs/auth-flow.md | 7 +++-- docs/site/src/content/docs/concepts.md | 9 +++++- 9 files changed, 165 insertions(+), 16 deletions(-) create mode 100755 dev/configure-operator.sh create mode 100644 dev/keycloak-route.yaml create mode 100755 dev/login-test.sh diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index 79e6c55..7b1326d 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -91,7 +91,7 @@ jobs: run: nebari-operator/dev/scripts/services/keycloak/setup.sh # ============================================================ - # Install nebari-operator + # Install nebari-operator, configured the way NIC configures it # ============================================================ - name: Install nebari-operator run: | @@ -102,6 +102,9 @@ jobs: kubectl wait --for=condition=available deployment/nebari-operator-controller-manager \ -n nebari-operator-system --timeout=120s + - name: Configure nebari-operator and expose Keycloak + run: dev/configure-operator.sh + - name: Label default namespace for operator run: kubectl label namespace default nebari.dev/managed=true --overwrite diff --git a/README.md b/README.md index b450a20..2d15a50 100644 --- a/README.md +++ b/README.md @@ -535,7 +535,9 @@ The `dev/` directory provides a Makefile for local development with creates a kind cluster with the full Nebari infrastructure stack - MetalLB, Envoy Gateway, cert-manager, Keycloak, and the nebari-operator - so every example deploys with NebariApp enabled, routing, TLS, and authentication -working just like a real Nebari cluster. +working just like a real Nebari cluster. The operator is configured the way NIC +configures it, and Keycloak is exposed at `keycloak.nebari.local`, so the full +login works locally (user `admin`, password `nebari-admin`). The first `make up-*` run takes ~5-10 minutes (cluster and infrastructure setup). Subsequent runs reuse the existing cluster and are fast. @@ -558,7 +560,10 @@ make up-podinfo # Deploy FastAPI Helm example (auth enabled, uses pre-built GHCR image) make up-fastapi -# Update /etc/hosts with NebariApp hostnames +# Log in to the FastAPI example with curl and check the app verified the token +make login-test + +# Update /etc/hosts with NebariApp and Keycloak hostnames make update-hosts # Delete the kind cluster @@ -566,8 +571,14 @@ make down ``` Each `up-*` target deploys with NebariApp enabled at `https://my-pack.nebari.local`, -waits for the NebariApp Ready condition, and updates `/etc/hosts` so you can access -the app in your browser. +checks that the app is actually served through the Gateway (`RoutingReady`, +`TLSReady` and a real HTTPS request, not just `Ready`), and updates `/etc/hosts` so +you can access the app in your browser. Run `make update-hosts` once to add +`keycloak.nebari.local` too, so the browser can reach the login page. The dev CA is +self-signed, so expect a certificate warning. + +The `/etc/hosts` steps use `sudo`. Everything else works without it: +`make login-test` and `dev/verify-nebariapp.sh` resolve hostnames themselves. ### What's not included @@ -886,6 +897,20 @@ naming convention (usually `-`). ``` Look for `AuthReady` condition. +### Every path returns 500 with auth enabled + +Envoy Gateway rejected the generated SecurityPolicy. Check its status: + +```bash +kubectl get securitypolicy my-pack-security -n my-pack \ + -o jsonpath='{range .status.ancestors[*].conditions[*]}{.type}={.status}: {.message}{"\n"}{end}' +``` + +`OIDC: error fetching endpoints from issuer` means the operator is pointing Envoy +Gateway at the wrong in-cluster Keycloak URL. Set `KEYCLOAK_ISSUER_SERVICE_PORT` and +`KEYCLOAK_ISSUER_CONTEXT_PATH` on the operator Deployment to match your Keycloak +Service (NIC sets these; `dev/configure-operator.sh` shows the kind equivalent). + ### TLS certificate not provisioning 1. Check cert-manager is running: diff --git a/dev/Makefile b/dev/Makefile index 05d763c..4b18bda 100644 --- a/dev/Makefile +++ b/dev/Makefile @@ -4,7 +4,7 @@ # (MetalLB, Envoy Gateway, cert-manager, Keycloak, nebari-operator) # so all examples deploy with NebariApp enabled. # -# Prerequisites: docker, kind, helm, kubectl, git +# Prerequisites: docker, kind, helm, kubectl, git, curl # # Usage: # make up-vanilla Deploy the vanilla YAML example @@ -12,7 +12,8 @@ # make up-basic Deploy the basic nginx Helm example # make up-fastapi Deploy the FastAPI Helm example (pre-built GHCR image) # make up-podinfo Deploy the podinfo wrapper Helm example -# make update-hosts Update /etc/hosts with NebariApp hostnames +# make update-hosts Update /etc/hosts with NebariApp and Keycloak hostnames +# make login-test Log in to the FastAPI example with curl # make down Delete the kind cluster VANILLA_DIR := ../examples/vanilla-yaml @@ -49,7 +50,7 @@ spec: endef export METALLB_POOL -.PHONY: up-vanilla up-kustomize up-basic up-fastapi up-podinfo down cluster update-hosts +.PHONY: up-vanilla up-kustomize up-basic up-fastapi up-podinfo down cluster update-hosts login-test # -------------------------------------------------------------------------- # cluster - create kind cluster with full Nebari infrastructure @@ -61,7 +62,7 @@ cluster: $(MAKE) _cluster-create; \ fi -_cluster-create: _clone-operator _kind-create _metallb _services _keycloak-setup _operator _label-namespace +_cluster-create: _clone-operator _kind-create _metallb _services _keycloak-setup _operator _configure-operator _label-namespace @echo "" @echo "==========================================" @echo "Cluster setup complete!" @@ -116,6 +117,11 @@ _operator: -n nebari-operator-system --timeout=120s @echo "nebari-operator is running" +# Set the operator env that NIC sets on real clusters and expose Keycloak at +# keycloak.nebari.local. See configure-operator.sh for why each one matters. +_configure-operator: + ./configure-operator.sh + _label-namespace: kubectl label namespace default nebari.dev/managed=true --overwrite @@ -179,6 +185,7 @@ up-fastapi: cluster @echo "FastAPI app deployed with NebariApp (auth enabled)." @echo " https://$(HOSTNAME)" @echo " Keycloak login: admin / nebari-admin" + @echo " Check the login flow from the terminal: make login-test" # -------------------------------------------------------------------------- # up-podinfo - deploy podinfo wrapper Helm example with NebariApp @@ -200,6 +207,14 @@ up-podinfo: cluster # -------------------------------------------------------------------------- update-hosts: CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh + CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh keycloak + +# -------------------------------------------------------------------------- +# login-test - log in to the FastAPI example with curl and check that the app +# verified the token. Run after up-fastapi; needs no /etc/hosts entries. +# -------------------------------------------------------------------------- +login-test: + ./login-test.sh $(HOSTNAME) # -------------------------------------------------------------------------- # down - delete the kind cluster diff --git a/dev/configure-operator.sh b/dev/configure-operator.sh new file mode 100755 index 0000000..cab12a7 --- /dev/null +++ b/dev/configure-operator.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Configures the nebari-operator in the kind dev cluster the way NIC configures +# it on a real Nebari cluster (see nebari-infrastructure-core +# pkg/argocd/templates/manifests/nebari-operator/deployment-patch.yaml), and +# exposes Keycloak through the Gateway at keycloak.nebari.local. +# +# The release install.yaml leaves these unset. Without them: +# - the in-cluster issuer defaults to port 8080 with no /auth path, Envoy +# Gateway rejects every SecurityPolicy, and auth-enabled apps return 500 +# - the browser is sent to in-cluster Keycloak URLs it cannot reach +# - no per-app Certificate is issued (TLSReady=False/ClusterIssuerNotConfigured) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +kubectl apply -f "${SCRIPT_DIR}/keycloak-route.yaml" + +kubectl set env -n nebari-operator-system deployment/nebari-operator-controller-manager \ + KEYCLOAK_ISSUER_SERVICE_PORT=80 \ + KEYCLOAK_ISSUER_CONTEXT_PATH=/auth \ + KEYCLOAK_EXTERNAL_URL=https://keycloak.nebari.local/auth \ + TLS_CLUSTER_ISSUER_NAME=nebari-ca-issuer + +kubectl rollout status -n nebari-operator-system \ + deployment/nebari-operator-controller-manager --timeout=120s diff --git a/dev/keycloak-route.yaml b/dev/keycloak-route.yaml new file mode 100644 index 0000000..9c80ecf --- /dev/null +++ b/dev/keycloak-route.yaml @@ -0,0 +1,31 @@ +# Exposes the dev Keycloak at https://keycloak.nebari.local/auth so the browser +# half of the OIDC login can reach it, the same way NIC exposes keycloak.. +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: keycloak + namespace: keycloak +spec: + parentRefs: + - name: nebari-gateway + namespace: envoy-gateway-system + sectionName: https + hostnames: + - keycloak.nebari.local + rules: + - matches: + - path: + type: PathPrefix + value: /auth + filters: + # The dev Keycloak trusts only the RFC 7239 Forwarded header + # (KC_PROXY_HEADERS=forwarded). Without it, Keycloak builds http:// + # login URLs and the browser leaves HTTPS mid-login. + - type: RequestHeaderModifier + requestHeaderModifier: + set: + - name: Forwarded + value: proto=https;host=keycloak.nebari.local + backendRefs: + - name: keycloak-keycloakx-http + port: 80 diff --git a/dev/login-test.sh b/dev/login-test.sh new file mode 100755 index 0000000..c23d9de --- /dev/null +++ b/dev/login-test.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Drives the browser half of the OIDC login with curl, against the local kind +# cluster, and prints what the app shows afterwards. +# +# ./login-test.sh [username] [password] +# +# Requests are pinned to the Gateway's LoadBalancer IP with --resolve, so this +# works without /etc/hosts entries. -k is used because the dev CA is self-signed. +set -euo pipefail + +APP_HOST=${1:?usage: login-test.sh [username] [password]} +USERNAME=${2:-admin} +PASSWORD=${3:-nebari-admin} +KC_HOST=keycloak.nebari.local + +GW=$(kubectl get svc -n envoy-gateway-system \ + -l gateway.envoyproxy.io/owning-gateway-name=nebari-gateway \ + -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}') +JAR=$(mktemp) +trap 'rm -f "$JAR"' EXIT +CURL=(curl -sk -b "$JAR" -c "$JAR" --resolve "$APP_HOST:443:$GW" --resolve "$KC_HOST:443:$GW") + +# 1. App redirects to Keycloak; follow it to the login form. +form=$("${CURL[@]}" -L "https://$APP_HOST/") +action=$(printf '%s' "$form" | grep -o 'action="[^"]*"' | head -1 | sed 's/^action="//; s/"$//; s/&/\&/g') +if [ -z "$action" ]; then + echo "FAIL: no Keycloak login form (is the SecurityPolicy accepted?)" >&2 + exit 1 +fi + +# 2. Submit credentials. Keycloak redirects to /oauth2/callback, Envoy sets the +# session cookies and redirects back to the app. +page=$("${CURL[@]}" -L --data-urlencode "username=$USERNAME" --data-urlencode "password=$PASSWORD" "$action") + +if printf '%s' "$page" | grep -q "Authenticated User"; then + echo "OK: logged in; app shows verified identity for '$USERNAME'" +else + echo "FAIL: app did not show a verified identity" >&2 + printf '%s' "$page" | tr -s ' \n' ' ' | grep -o '

Not Authenticated

[^<]*' | sed 's/<[^>]*>//g' >&2 + exit 1 +fi diff --git a/docs/auth-flow.md b/docs/auth-flow.md index 5c502cc..e411451 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -394,9 +394,10 @@ for the full set of OIDC fields. ## Limitations -- **Local development:** The OIDC flow requires Keycloak and Envoy Gateway. When - developing locally with kind, set `nebariapp.enabled=false` and test without auth. - The FastAPI example shows "Not Authenticated" when no IdToken cookie is present. +- **Local development:** The `dev/` Makefile builds a kind cluster with Keycloak, Envoy + Gateway, cert-manager and the operator, so you can test the full login flow locally: + `cd dev && make up-fastapi`, then log in with the Keycloak credentials it prints. The + FastAPI example shows "Not Authenticated" when no valid IdToken cookie is present. - **Token expiration:** Envoy Gateway handles token refresh automatically via refresh tokens stored in cookies. Your app does not need to handle token refresh. diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index f687387..b396f72 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -381,9 +381,10 @@ for the full set of OIDC fields. ## Limitations -- **Local development:** The OIDC flow requires Keycloak and Envoy Gateway. When - developing locally with kind, set `nebariapp.enabled=false` and test without auth. - The FastAPI example shows "Not Authenticated" when no IdToken cookie is present. +- **Local development:** The `dev/` Makefile builds a kind cluster with Keycloak, Envoy + Gateway, cert-manager and the operator, so you can test the full login flow locally: + `cd dev && make up-fastapi`, then log in with the Keycloak credentials it prints. The + FastAPI example shows "Not Authenticated" when no valid IdToken cookie is present. - **Token expiration:** Envoy Gateway handles token refresh automatically via refresh tokens stored in cookies. Your app does not need to handle token refresh. diff --git a/docs/site/src/content/docs/concepts.md b/docs/site/src/content/docs/concepts.md index 28db1bf..db4ab94 100644 --- a/docs/site/src/content/docs/concepts.md +++ b/docs/site/src/content/docs/concepts.md @@ -155,11 +155,18 @@ cd dev make up-vanilla # deploy vanilla YAML example make up-kustomize # deploy kustomize example (dev overlay) make up-basic # deploy Helm nginx example +make up-podinfo # deploy the podinfo wrapper Helm example make up-fastapi # deploy FastAPI Helm example (auth enabled) -make update-hosts # update /etc/hosts with NebariApp hostnames +make login-test # log in to the FastAPI example with curl +make update-hosts # update /etc/hosts with NebariApp and Keycloak hostnames make down # delete the kind cluster ``` +Each `up-*` target checks that the app is actually served through the Gateway, not just +that the NebariApp is `Ready`. The cluster configures the operator the way NIC does on a +real Nebari cluster and exposes Keycloak at `keycloak.nebari.local`, so the full OIDC login +works locally (user `admin`, password `nebari-admin`). + The first `make up-*` run takes 5-10 minutes (cluster and infrastructure setup). Subsequent runs reuse the existing cluster and are fast. From 634d28f3f7fb9bf0d59f3e6d15014763f5947c5e Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:17 +0200 Subject: [PATCH 03/16] Add routing to examples and docs, and check real traffic in CI (#34) With no spec.routing the operator creates no HTTPRoute and no TLS, yet the NebariApp reports Ready=True. basic-nginx and wrap-existing-chart shipped without routing, so they deployed cleanly and returned 404, and CI passed because it only waited for Ready. - basic-nginx and wrap-existing-chart values: add routing.routes and routing.tls (chart versions bumped) - Add routing to every NebariApp snippet in the README and docs, and say what happens without it - dev/verify-nebariapp.sh waits for RoutingReady and TLSReady (and AuthReady with --auth), checks the HTTPRoute and SecurityPolicy exist, and sends an HTTPS request through the Gateway - Makefile and integration CI use it instead of waiting on Ready. The Makefile waited on nebariapp/my-pack-my-pack, which never exists: the release name already contains the chart name, so the NebariApp is my-pack - lint.yaml fails if a chart renders a NebariApp without routing - wrap-existing-chart uses helm dependency build against its committed lock --- .github/workflows/lint.yaml | 15 ++++- .github/workflows/test-integration.yaml | 58 +++++++----------- .github/workflows/test.yaml | 4 +- README.md | 27 +++++++-- dev/Makefile | 8 +-- dev/verify-nebariapp.sh | 60 +++++++++++++++++++ docs/nebariapp-crd-reference.md | 7 ++- docs/site/src/content/docs/concepts.md | 10 ++++ .../content/docs/nebariapp-crd-reference.md | 7 ++- examples/basic-nginx/chart/Chart.yaml | 2 +- examples/basic-nginx/chart/values.yaml | 8 +++ examples/wrap-existing-chart/README.md | 4 +- examples/wrap-existing-chart/chart/Chart.yaml | 2 +- .../wrap-existing-chart/chart/values.yaml | 8 +++ 14 files changed, 165 insertions(+), 55 deletions(-) create mode 100755 dev/verify-nebariapp.sh diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 819105c..d05e8f3 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -65,8 +65,8 @@ jobs: run: helm template test examples/auth-fastapi/chart/ --set nebariapp.enabled=true --set nebariapp.hostname=test.example.com # Wrap existing chart (Podinfo) Helm example - - name: Update podinfo dependency - run: helm dependency update examples/wrap-existing-chart/chart/ + - name: Build wrap-existing-chart dependencies + run: helm dependency build examples/wrap-existing-chart/chart/ - name: Lint wrap-existing-chart run: helm lint examples/wrap-existing-chart/chart/ @@ -76,3 +76,14 @@ jobs: - name: Template wrap-existing-chart (NebariApp enabled) run: helm template test examples/wrap-existing-chart/chart/ --set nebariapp.enabled=true --set nebariapp.hostname=test.example.com + + # Without spec.routing the operator creates no HTTPRoute or TLS, yet the + # NebariApp still reports Ready. Fail fast if an example chart drops it. + - name: Every chart renders a NebariApp with routing + run: | + for c in basic-nginx auth-fastapi wrap-existing-chart; do + routes=$(helm template test examples/$c/chart/ --set nebariapp.enabled=true --set nebariapp.hostname=test.example.com \ + | yq 'select(.kind == "NebariApp") | .spec.routing.routes | length') + if [ "${routes:-0}" -lt 1 ]; then echo "$c: NebariApp has no spec.routing.routes"; exit 1; fi + echo "$c: $routes route(s)" + done diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index 7b1326d..d50180a 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -119,25 +119,25 @@ jobs: ghcr.io/nebari-dev/software-pack-template/auth-fastapi-example:latest \ --name pack-integration + # Each "Verify" step waits for RoutingReady, TLSReady (and AuthReady with + # --auth), checks the generated HTTPRoute/SecurityPolicy exist, and sends a + # request through the Gateway. Ready=True alone is not enough: a NebariApp + # without spec.routing is Ready and unreachable. + # ============================================================ # Test: vanilla-yaml # ============================================================ - name: "Deploy vanilla-yaml" run: | - # Apply deployment + service kubectl apply -f examples/vanilla-yaml/deployment.yaml \ -f examples/vanilla-yaml/service.yaml - - # Apply nebariapp with hostname patched for nebari.local domain sed 's/my-pack.nebari.example.com/vanilla.nebari.local/' \ examples/vanilla-yaml/nebariapp.yaml | kubectl apply -f - - - name: "Verify vanilla-yaml NebariApp" + - name: "Verify vanilla-yaml" run: | kubectl wait --for=condition=ready pod -l app=my-pack --timeout=120s - kubectl wait --for=condition=Ready nebariapp/my-pack --timeout=120s - kubectl get httproute -l app.kubernetes.io/managed-by=nebari-operator - echo "vanilla-yaml: NebariApp Ready" + dev/verify-nebariapp.sh my-pack vanilla.nebari.local - name: "Cleanup vanilla-yaml" if: always() @@ -150,22 +150,16 @@ jobs: # Test: kustomize-nginx (production overlay - auth enabled) # ============================================================ - name: "Deploy kustomize-nginx (production)" - run: | - kubectl apply -k examples/kustomize-nginx/overlays/production/ + run: kubectl apply -k examples/kustomize-nginx/overlays/production/ - - name: "Verify kustomize-nginx NebariApp" + - name: "Verify kustomize-nginx" run: | kubectl wait --for=condition=ready pod -l app=my-pack --timeout=120s - kubectl wait --for=condition=Ready nebariapp/my-pack --timeout=120s - kubectl get httproute -l app.kubernetes.io/managed-by=nebari-operator - # Production overlay has auth enabled - verify SecurityPolicy - kubectl get securitypolicy - echo "kustomize-nginx (production): NebariApp Ready with auth" + dev/verify-nebariapp.sh my-pack my-pack.nebari.example.com --auth - name: "Cleanup kustomize-nginx" if: always() - run: | - kubectl delete -k examples/kustomize-nginx/overlays/production/ --ignore-not-found --wait --timeout=60s + run: kubectl delete -k examples/kustomize-nginx/overlays/production/ --ignore-not-found --wait --timeout=60s # ============================================================ # Test: basic-nginx (Helm, no auth) @@ -178,12 +172,8 @@ jobs: --set nebariapp.hostname=basic.nebari.local \ --wait --timeout 2m - - name: "Verify basic-nginx NebariApp" - run: | - kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=test-basic --timeout=120s - kubectl wait --for=condition=Ready nebariapp/test-basic-my-pack --timeout=120s - kubectl get httproute -l app.kubernetes.io/managed-by=nebari-operator - echo "basic-nginx: NebariApp Ready" + - name: "Verify basic-nginx" + run: dev/verify-nebariapp.sh test-basic-my-pack basic.nebari.local - name: "Cleanup basic-nginx" if: always() @@ -201,14 +191,9 @@ jobs: --set nebariapp.auth.enabled=true \ --wait --timeout 2m - - name: "Verify auth-fastapi NebariApp" + - name: "Verify auth-fastapi" run: | - kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=test-fastapi --timeout=120s - kubectl wait --for=condition=Ready nebariapp/test-fastapi-my-pack --timeout=120s - kubectl get httproute -l app.kubernetes.io/managed-by=nebari-operator - # Auth enabled - verify SecurityPolicy - kubectl get securitypolicy - echo "auth-fastapi: NebariApp Ready with auth" + dev/verify-nebariapp.sh test-fastapi-my-pack fastapi.nebari.local --auth - name: "Cleanup auth-fastapi" if: always() @@ -219,20 +204,18 @@ jobs: # ============================================================ - name: "Deploy wrap-existing-chart (podinfo)" run: | - helm dependency update examples/wrap-existing-chart/chart/ + helm dependency build examples/wrap-existing-chart/chart/ helm install test-wrap examples/wrap-existing-chart/chart/ \ --set nebariapp.enabled=true \ --set nebariapp.hostname=podinfo.nebari.local \ --wait --timeout 3m - - name: "Verify wrap-existing-chart NebariApp" + - name: "Verify wrap-existing-chart" run: | # Podinfo's chart labels its pods with app.kubernetes.io/name=-podinfo # and does not set app.kubernetes.io/instance, so wait on the Deployment instead. kubectl wait --for=condition=available deployment/test-wrap-podinfo --timeout=120s - kubectl wait --for=condition=Ready nebariapp/test-wrap-my-pack --timeout=120s - kubectl get httproute -l app.kubernetes.io/managed-by=nebari-operator - echo "wrap-existing-chart: NebariApp Ready" + dev/verify-nebariapp.sh test-wrap-my-pack podinfo.nebari.local - name: "Cleanup wrap-existing-chart" if: always() @@ -254,12 +237,15 @@ jobs: kubectl get httproute -A || true echo "" echo "=== SecurityPolicies ===" - kubectl get securitypolicy -A || true + kubectl get securitypolicy -A -o yaml | grep -A6 'conditions:' || true echo "" echo "=== Operator logs ===" kubectl logs -n nebari-operator-system deployment/nebari-operator-controller-manager \ --tail=100 || true echo "" + echo "=== Envoy Gateway logs ===" + kubectl logs -n envoy-gateway-system deployment/envoy-gateway --tail=50 || true + echo "" echo "=== Pods ===" kubectl get pods -A || true echo "" diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 4c05392..7c9ff8f 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -116,8 +116,8 @@ jobs: run: helm uninstall test-fastapi -n test-fastapi --wait # Test podinfo wrapper Helm example - - name: Update podinfo dependency - run: helm dependency update examples/wrap-existing-chart/chart/ + - name: Build wrap-existing-chart dependencies + run: helm dependency build examples/wrap-existing-chart/chart/ - name: Install wrap-existing-chart run: | diff --git a/README.md b/README.md index 2d15a50..26835fe 100644 --- a/README.md +++ b/README.md @@ -175,6 +175,10 @@ software-pack-template/ README.md dev/ Makefile # Local dev with full Nebari stack on kind + configure-operator.sh # Configures the operator the way NIC does + keycloak-route.yaml # Exposes Keycloak at keycloak.nebari.local + verify-nebariapp.sh # Checks an app is served, not just Ready + login-test.sh # Logs in with curl and checks the app .cache/ # (gitignored) Cloned nebari-operator scripts docs/ nebariapp-crd-reference.md # Full NebariApp field reference @@ -208,7 +212,8 @@ spec: name: my-pack # Service name in the same namespace port: 80 # Service port (1-65535) - # Optional: path-based routing rules + # Routing: required in practice. Without it the operator creates no + # HTTPRoute and no TLS, yet the NebariApp still reports Ready. routing: routes: - pathPrefix: / # Match all paths (default behavior) @@ -263,6 +268,11 @@ To render the NebariApp in a Helm chart with the official service: name: '{{ include "my-pack.fullname" . | toJson }}' port: '{{ .Values.service.port }}' + routing: + routes: + - pathPrefix: / + tls: + enabled: true ``` 3. Render it in `templates/nebariapp.yaml`. The `if` makes the NebariApp optional, @@ -469,13 +479,18 @@ nebariapp: service: name: '{{ printf "%s-podinfo" .Release.Name | toJson }}' # Upstream service port: 9898 + routing: + routes: + - pathPrefix: / + tls: + enabled: true ``` **You don't rewrite the app. You just connect it to Nebari.** ```bash # Build dependencies -helm dependency update examples/wrap-existing-chart/chart/ +helm dependency build examples/wrap-existing-chart/chart/ # Deploy standalone helm install test-wrap examples/wrap-existing-chart/chart/ @@ -597,6 +612,7 @@ Runs on every push and PR. Validates all examples: - `kubectl apply --dry-run=client` for the vanilla YAML example - `kubectl kustomize` for each Kustomize overlay - `helm lint` and `helm template` for each Helm chart (both NebariApp enabled and disabled) +- Fails if any example chart renders a NebariApp without `spec.routing` ### Build Images (`build-images.yaml`) @@ -791,7 +807,8 @@ kubectl describe nebariapp my-pack -n my-pack # RoutingReady: True - HTTPRoute created # TLSReady: True - Certificate provisioned # AuthReady: True - SecurityPolicy created (if auth enabled) -# Ready: True - All components ready +# Ready: True - Core checks passed. Does NOT wait for the others: +# a NebariApp with no routing is Ready and unreachable. ``` ## Customizing for Your Own Application @@ -943,13 +960,13 @@ The Helm examples depend on the `nebari-app` chart. Fetch it before installing: helm dependency build examples/basic-nginx/chart/ ``` -### `helm dependency update` fails +### `helm dependency build` fails The dependencies are pulled from OCI registries, so ensure Helm 3.8+ is installed: ```bash helm version -helm dependency update examples/wrap-existing-chart/chart/ +helm dependency build examples/wrap-existing-chart/chart/ ``` ## Documentation Portal diff --git a/dev/Makefile b/dev/Makefile index 4b18bda..595b4c9 100644 --- a/dev/Makefile +++ b/dev/Makefile @@ -134,7 +134,7 @@ up-vanilla: cluster sed 's/my-pack.nebari.example.com/$(HOSTNAME)/' \ $(VANILLA_DIR)/nebariapp.yaml | kubectl apply -f - kubectl wait --for=condition=ready pod -l app=my-pack --timeout=120s - kubectl wait --for=condition=Ready nebariapp/my-pack --timeout=120s + ./verify-nebariapp.sh my-pack $(HOSTNAME) CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh @echo "" @echo "Vanilla nginx deployed with NebariApp." @@ -148,7 +148,7 @@ up-kustomize: cluster | sed 's/my-pack.dev.nebari.example.com/$(HOSTNAME)/' \ | kubectl apply -f - kubectl wait --for=condition=ready pod -l app=my-pack --timeout=120s - kubectl wait --for=condition=Ready nebariapp/my-pack --timeout=120s + ./verify-nebariapp.sh my-pack $(HOSTNAME) CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh @echo "" @echo "Kustomize nginx deployed with NebariApp." @@ -163,7 +163,7 @@ up-basic: cluster --set nebariapp.enabled=true \ --set nebariapp.hostname=$(HOSTNAME) \ --wait --timeout 2m - kubectl wait --for=condition=Ready nebariapp/my-pack-my-pack --timeout=120s + ./verify-nebariapp.sh my-pack $(HOSTNAME) CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh @echo "" @echo "Basic nginx deployed with NebariApp." @@ -196,7 +196,7 @@ up-podinfo: cluster --set nebariapp.enabled=true \ --set nebariapp.hostname=$(HOSTNAME) \ --wait --timeout 3m - kubectl wait --for=condition=Ready nebariapp/my-pack-my-pack --timeout=120s + ./verify-nebariapp.sh my-pack $(HOSTNAME) CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh @echo "" @echo "Podinfo deployed with NebariApp." diff --git a/dev/verify-nebariapp.sh b/dev/verify-nebariapp.sh new file mode 100755 index 0000000..f4fb745 --- /dev/null +++ b/dev/verify-nebariapp.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Checks that a NebariApp is actually serving traffic, not just reporting Ready. +# +# ./verify-nebariapp.sh [--auth] [--namespace ] +# +# Ready=True alone proves little: a NebariApp with no spec.routing is Ready but +# has no HTTPRoute and returns 404. This waits for the routing, TLS and (with +# --auth) auth conditions, checks the generated resources exist, and sends a +# request through the Gateway: 200 without auth, a redirect to Keycloak with it. +set -euo pipefail + +NAME=${1:?usage: verify-nebariapp.sh [--auth] [--namespace ]} +HOST=${2:?usage: verify-nebariapp.sh [--auth] [--namespace ]} +shift 2 +AUTH=false +NS=default +while [ $# -gt 0 ]; do + case "$1" in + --auth) AUTH=true ;; + --namespace) NS=$2; shift ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac + shift +done + +wait_cond() { + kubectl wait -n "$NS" --for=condition="$1" "nebariapp/$NAME" --timeout=180s +} + +wait_cond Ready +wait_cond RoutingReady +wait_cond TLSReady +kubectl get -n "$NS" httproute "$NAME-route" >/dev/null + +if $AUTH; then + wait_cond AuthReady + kubectl get -n "$NS" securitypolicy "$NAME-security" >/dev/null + expect="302 https://keycloak.nebari.local/" +else + expect="200 " +fi + +GW=$(kubectl get svc -n envoy-gateway-system \ + -l gateway.envoyproxy.io/owning-gateway-name=nebari-gateway \ + -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}') + +# Envoy Gateway needs a few seconds to push new listeners and policies. +for _ in $(seq 1 30); do + got=$(curl -sk -o /dev/null -w '%{http_code} %{redirect_url}' \ + --resolve "$HOST:443:$GW" "https://$HOST/" || true) + case "$got" in + "$expect"*) echo "OK: $NAME serves https://$HOST/ ($got)"; exit 0 ;; + esac + sleep 4 +done + +echo "FAIL: https://$HOST/ returned '$got', expected '$expect...'" >&2 +kubectl get -n "$NS" nebariapp "$NAME" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status}/{.reason}: {.message}{"\n"}{end}' >&2 +exit 1 diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 32b7a5b..8869544 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -62,7 +62,7 @@ spec: |-------|------|----------|---------|-------------| | `hostname` | string | Yes | - | FQDN where the app will be accessible. Used to generate the HTTPRoute and TLS certificate. Must match pattern `^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`. | | `service` | [ServiceReference](#specservice) | Yes | - | The backend Kubernetes Service that receives traffic. | -| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. | +| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | | `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | | `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | | `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. The operator scopes RBAC on the OIDC client Secret to this ServiceAccount, so only the app's pods can read its credentials. | @@ -263,6 +263,11 @@ spec: service: name: my-pack port: 80 + routing: + routes: + - pathPrefix: / + tls: + enabled: true ``` When deploying standalone (without Nebari), skip this file in your `kubectl apply`. diff --git a/docs/site/src/content/docs/concepts.md b/docs/site/src/content/docs/concepts.md index db4ab94..02b1d24 100644 --- a/docs/site/src/content/docs/concepts.md +++ b/docs/site/src/content/docs/concepts.md @@ -23,6 +23,7 @@ You typically make it conditional so the chart works both standalone and on Neba "metadata" (dict "name" (include "my-pack.fullname" .) "namespace" .Release.Namespace + "labels" (include "my-pack.labels" . | fromYaml) ) "spec" (omit .Values.nebariapp "enabled") "tplCtx" . @@ -33,6 +34,10 @@ You typically make it conditional so the chart works both standalone and on Neba With plain YAML or Kustomize, the NebariApp manifest is always present. When deploying standalone, skip that file or exclude it from your apply command. +Whichever method you use, give the NebariApp a `spec.routing` block with at least one +route. Without it the operator creates no HTTPRoute and no TLS certificate, and the +NebariApp still reports `Ready`, so the pack deploys cleanly and is unreachable. + ## Deployment methods All three Kubernetes deployment methods are first-class in the template and in ArgoCD. @@ -92,6 +97,11 @@ nebariapp: service: name: '{{ printf "%s-podinfo" .Release.Name | toJson }}' # upstream service name port: 9898 + routing: + routes: + - pathPrefix: / + tls: + enabled: true ``` `templates/nebariapp.yaml` is the same short `include` shown above. Run diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 9d569b4..ef91f76 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -64,7 +64,7 @@ spec: |-------|------|----------|---------|-------------| | `hostname` | string | Yes | - | FQDN where the app will be accessible. Used to generate the HTTPRoute and TLS certificate. Must match pattern `^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`. | | `service` | [ServiceReference](#specservice) | Yes | - | The backend Kubernetes Service that receives traffic. | -| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. | +| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | | `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | | `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | | `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. The operator scopes RBAC on the OIDC client Secret to this ServiceAccount, so only the app's pods can read its credentials. | @@ -267,6 +267,11 @@ spec: service: name: my-pack port: 80 + routing: + routes: + - pathPrefix: / + tls: + enabled: true ``` When deploying standalone (without Nebari), skip this file in your `kubectl apply`. diff --git a/examples/basic-nginx/chart/Chart.yaml b/examples/basic-nginx/chart/Chart.yaml index dcbbcaa..101849e 100644 --- a/examples/basic-nginx/chart/Chart.yaml +++ b/examples/basic-nginx/chart/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: my-pack description: A Nebari Software Pack (replace with your description) type: application -version: 0.1.1 +version: 0.1.2 appVersion: "1.0.0" dependencies: - name: nebari-app diff --git a/examples/basic-nginx/chart/values.yaml b/examples/basic-nginx/chart/values.yaml index 438bc53..65c257b 100644 --- a/examples/basic-nginx/chart/values.yaml +++ b/examples/basic-nginx/chart/values.yaml @@ -35,6 +35,14 @@ nebariapp: service: name: '{{ include "my-pack.fullname" . | toJson }}' port: '{{ .Values.service.port }}' + # Without a routing block the operator creates no HTTPRoute and no TLS + # certificate, and the NebariApp still reports Ready. Keep at least one route. + routing: + routes: + - pathPrefix: / + pathType: PathPrefix + tls: + enabled: true auth: enabled: false provider: keycloak diff --git a/examples/wrap-existing-chart/README.md b/examples/wrap-existing-chart/README.md index e092405..3bc127c 100644 --- a/examples/wrap-existing-chart/README.md +++ b/examples/wrap-existing-chart/README.md @@ -88,7 +88,7 @@ To override upstream values via ArgoCD: ```bash # Build dependencies first -helm dependency update ./chart/ +helm dependency build ./chart/ # Deploy on Nebari helm install my-pack ./chart/ \ @@ -105,7 +105,7 @@ helm install my-pack ./chart/ \ ## Local development (standalone, no Nebari) ```bash -helm dependency update ./chart/ +helm dependency build ./chart/ helm install test-wrap ./chart/ # Access via port-forward diff --git a/examples/wrap-existing-chart/chart/Chart.yaml b/examples/wrap-existing-chart/chart/Chart.yaml index 5041671..8a123ff 100644 --- a/examples/wrap-existing-chart/chart/Chart.yaml +++ b/examples/wrap-existing-chart/chart/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: my-pack description: A Nebari Software Pack wrapping Podinfo type: application -version: 0.1.1 +version: 0.1.2 appVersion: "6.10.1" dependencies: - name: nebari-app diff --git a/examples/wrap-existing-chart/chart/values.yaml b/examples/wrap-existing-chart/chart/values.yaml index ffd7246..bb2edf9 100644 --- a/examples/wrap-existing-chart/chart/values.yaml +++ b/examples/wrap-existing-chart/chart/values.yaml @@ -14,6 +14,14 @@ nebariapp: # Points to the upstream podinfo chart's service name: '{{ include "my-pack.podinfo-service-name" . | toJson }}' port: 9898 + # Without a routing block the operator creates no HTTPRoute and no TLS + # certificate, and the NebariApp still reports Ready. Keep at least one route. + routing: + routes: + - pathPrefix: / + pathType: PathPrefix + tls: + enabled: true auth: enabled: false provider: keycloak From 83e2a414ac2d93061a1b7e2cd287295cc5a6b550 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:17 +0200 Subject: [PATCH 04/16] Make ArgoCD examples deployable on NIC clusters (#35) NIC locks ArgoCD's default AppProject to deny-all and expects packs in nebari-apps. Every Application example used project: default, and their CreateNamespace=true namespaces were never labeled for the operator. - All Application examples: project: nebari-apps, plus managedNamespaceMetadata labeling the namespace nebari.dev/managed=true - build-your-own: a complete Application example, the NIC requirements, how to check the pack is reachable, and the repository Secret needed for private sources - CRD reference: label the namespace through ArgoCD - lint.yaml fails if an example uses project: default --- .github/workflows/lint.yaml | 7 ++ README.md | 22 +++++- docs/nebariapp-crd-reference.md | 20 +++++- docs/site/src/content/docs/build-your-own.md | 68 ++++++++++++++++++- .../content/docs/nebariapp-crd-reference.md | 16 +++++ examples/auth-fastapi/README.md | 7 +- examples/basic-nginx/README.md | 7 +- examples/kustomize-nginx/README.md | 7 +- examples/vanilla-yaml/README.md | 7 +- examples/wrap-existing-chart/README.md | 7 +- 10 files changed, 158 insertions(+), 10 deletions(-) diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index d05e8f3..4f9cede 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -87,3 +87,10 @@ jobs: if [ "${routes:-0}" -lt 1 ]; then echo "$c: NebariApp has no spec.routing.routes"; exit 1; fi echo "$c: $routes route(s)" done + + # NIC locks the "default" AppProject to deny-all; packs must use nebari-apps. + - name: ArgoCD examples use the nebari-apps project + run: | + if grep -rn --include='*.md' -E '^\s*project:\s*default\b' README.md docs examples; then + echo "ArgoCD Application examples must use 'project: nebari-apps'"; exit 1 + fi diff --git a/README.md b/README.md index 26835fe..d7df872 100644 --- a/README.md +++ b/README.md @@ -613,6 +613,7 @@ Runs on every push and PR. Validates all examples: - `kubectl kustomize` for each Kustomize overlay - `helm lint` and `helm template` for each Helm chart (both NebariApp enabled and disabled) - Fails if any example chart renders a NebariApp without `spec.routing` +- Fails if any ArgoCD Application example uses `project: default` (deny-all on NIC) ### Build Images (`build-images.yaml`) @@ -691,7 +692,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -712,6 +713,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` **ArgoCD with Kustomize:** @@ -723,7 +729,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -738,6 +744,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` **ArgoCD with plain YAML (directory):** @@ -749,7 +760,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -765,6 +776,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` ### Option B: kubectl apply (plain YAML) diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 8869544..19001b4 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -242,7 +242,25 @@ The namespace containing the NebariApp must be labeled for the operator to proce kubectl label namespace my-pack nebari.dev/managed=true ``` -Without this label, the NebariApp will show `NamespaceNotOptedIn` and no resources will be created. +Without this label, the NebariApp will show `NamespaceNotOptedIn` and no resources +will be created. + +When ArgoCD creates the namespace (`CreateNamespace=true`), have it apply the label too, +so nobody has to run `kubectl` by hand: + +```yaml +spec: + syncPolicy: + syncOptions: + - CreateNamespace=true + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" +``` + +ArgoCD only applies `managedNamespaceMetadata` to a namespace that the same Application +creates. If the namespace already exists, label it yourself. + ## Deployment Patterns diff --git a/docs/site/src/content/docs/build-your-own.md b/docs/site/src/content/docs/build-your-own.md index 9030405..9864d6d 100644 --- a/docs/site/src/content/docs/build-your-own.md +++ b/docs/site/src/content/docs/build-your-own.md @@ -42,9 +42,75 @@ ArgoCD then: This GitOps approach means the pack configuration is version-controlled alongside everything else in your infrastructure. +On a cluster built with Nebari Infrastructure Core (NIC), the Application needs two things the generic ArgoCD docs won't tell you: + +- **`project: nebari-apps`.** NIC creates this AppProject for packs. Its built-in `default` project is locked to deny-all, so an Application left on `default` never syncs. +- **The namespace opt-in label.** The operator ignores NebariApps in namespaces without `nebari.dev/managed=true` (status `NamespaceNotOptedIn`). When ArgoCD creates the namespace, have it apply the label with `managedNamespaceMetadata`. ArgoCD only does this for a namespace the same Application creates, so label a pre-existing namespace yourself. + +A Helm pack from a git repo looks like this: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: my-pack + namespace: argocd +spec: + project: nebari-apps + source: + repoURL: https://github.com/my-org/my-pack.git + targetRevision: main + path: chart + helm: + valuesObject: + nebariapp: + enabled: true + hostname: my-pack.nebari.example.com + destination: + server: https://kubernetes.default.svc + namespace: my-pack + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" +``` + +Once it syncs, check that the pack is actually reachable, not just `Ready`: + +```bash +kubectl get nebariapp -n my-pack my-pack \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status}/{.reason}{"\n"}{end}' +``` + +`RoutingReady` must be `True`. `Ready=True` alone is not enough: a NebariApp without `spec.routing` reports `Ready` and serves nothing. + ## Private and organizational packs -Packs do not need to be public. Put yours in a private GitHub repo, an internal Git host, or any other location ArgoCD can reach. It works the same as a published pack - it just won't appear in the public catalog. +Packs do not need to be public. Put yours in a private GitHub repo, an internal Git host, or a private chart registry. The `nebari-apps` project accepts any source, so nothing changes in the Application itself, and the pack won't appear in the public catalog. + +ArgoCD does need credentials to read a private source, and NIC does not create them. Add a repository Secret in the `argocd` namespace (see the [ArgoCD repository docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#repositories)): + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: my-pack-repo + namespace: argocd + labels: + argocd.argoproj.io/secret-type: repository +stringData: + type: git + url: https://github.com/my-org/my-pack.git + username: git # any non-empty value works with a GitHub token + password: +``` + +For a private OCI chart registry, use `type: helm`, `enableOCI: "true"`, and the registry host and path without the `oci://` prefix as `url`. Keep the token out of the gitops repo in plain text; create the Secret with your secrets tooling (Sealed Secrets, External Secrets, or `kubectl` directly). ## Going deeper diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index ef91f76..376502a 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -248,6 +248,22 @@ kubectl label namespace my-pack nebari.dev/managed=true Without this label, the NebariApp will show `NamespaceNotOptedIn` and no resources will be created. +When ArgoCD creates the namespace (`CreateNamespace=true`), have it apply the label too, +so nobody has to run `kubectl` by hand: + +```yaml +spec: + syncPolicy: + syncOptions: + - CreateNamespace=true + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" +``` + +ArgoCD only applies `managedNamespaceMetadata` to a namespace that the same Application +creates. If the namespace already exists, label it yourself. + ## Deployment Patterns The NebariApp resource can be included in your pack using any deployment method. diff --git a/examples/auth-fastapi/README.md b/examples/auth-fastapi/README.md index eb2b7ec..dd17dd1 100644 --- a/examples/auth-fastapi/README.md +++ b/examples/auth-fastapi/README.md @@ -40,7 +40,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -61,6 +61,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` ### Helm install diff --git a/examples/basic-nginx/README.md b/examples/basic-nginx/README.md index 2a04af5..890af08 100644 --- a/examples/basic-nginx/README.md +++ b/examples/basic-nginx/README.md @@ -22,7 +22,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -41,6 +41,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` The nebari-operator will automatically create: diff --git a/examples/kustomize-nginx/README.md b/examples/kustomize-nginx/README.md index 0ebe116..89f22cb 100644 --- a/examples/kustomize-nginx/README.md +++ b/examples/kustomize-nginx/README.md @@ -49,7 +49,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -63,6 +63,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` ### kubectl apply -k diff --git a/examples/vanilla-yaml/README.md b/examples/vanilla-yaml/README.md index 8b99357..1ae7dea 100644 --- a/examples/vanilla-yaml/README.md +++ b/examples/vanilla-yaml/README.md @@ -37,7 +37,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -53,6 +53,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` Before deploying, edit `nebariapp.yaml` and set `hostname` to your actual diff --git a/examples/wrap-existing-chart/README.md b/examples/wrap-existing-chart/README.md index 3bc127c..ededb23 100644 --- a/examples/wrap-existing-chart/README.md +++ b/examples/wrap-existing-chart/README.md @@ -50,7 +50,7 @@ metadata: name: my-pack namespace: argocd spec: - project: default + project: nebari-apps # NIC's AppProject for packs; "default" is deny-all source: repoURL: https://github.com/YOUR-ORG/YOUR-REPO.git targetRevision: main @@ -69,6 +69,11 @@ spec: selfHeal: true syncOptions: - CreateNamespace=true + # The operator only reconciles NebariApps in namespaces labeled + # nebari.dev/managed=true. ArgoCD applies this to the namespace it creates. + managedNamespaceMetadata: + labels: + nebari.dev/managed: "true" ``` To override upstream values via ArgoCD: From 464c5790f9601c1fd0a75f080ed7489517d2b2a0 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:17 +0200 Subject: [PATCH 05/16] Stop documenting auth.groups as access control (#36) Operator v0.1.1 creates the groups in Keycloak and lists them for the landing page, but the SecurityPolicy it generates has no authorization rule, so any user in the realm gets through (nebari-dev/nebari-operator#153). - CRD reference, README and example READMEs say groups is not enforced and point to checking the groups claim in the app - Drop "restricted to groups" wording from the kustomize production overlay docs and comment the overlay's groups list --- README.md | 23 ++++++++++--------- docs/nebariapp-crd-reference.md | 7 +++--- docs/site/src/content/docs/concepts.md | 2 +- .../content/docs/nebariapp-crd-reference.md | 7 +++--- examples/kustomize-nginx/README.md | 4 ++-- .../overlays/production/nebariapp-patch.yaml | 2 ++ examples/vanilla-yaml/README.md | 14 ++++------- 7 files changed, 30 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index d7df872..5dc2807 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ software-pack-template/ dev/ # Dev overlay: dev hostname, no auth kustomization.yaml nebariapp-patch.yaml - production/ # Prod overlay: prod hostname, auth + groups + production/ # Prod overlay: prod hostname, auth enabled kustomization.yaml nebariapp-patch.yaml README.md @@ -231,7 +231,7 @@ spec: - openid - profile - email - groups: # Restrict to specific groups (optional) + groups: # NOT enforced by the gateway in v0.1.1; see below - admin enforceAtGateway: true # Create SecurityPolicy at gateway (default: true) @@ -372,7 +372,7 @@ kubectl kustomize examples/kustomize-nginx/overlays/dev/ # Deploy the dev overlay on Nebari kubectl apply -k examples/kustomize-nginx/overlays/dev/ -# Deploy the production overlay (auth enabled, group-restricted) +# Deploy the production overlay (auth enabled) kubectl apply -k examples/kustomize-nginx/overlays/production/ ``` @@ -886,14 +886,15 @@ routing: ### Restricting access to specific groups -```yaml -# In the NebariApp spec (any deployment method) -auth: - enabled: true - groups: - - admin - - data-science-team -``` +`auth.groups` does **not** restrict access in operator v0.1.1. The operator +creates the groups in Keycloak and lists them on the landing page, but the +SecurityPolicy it generates has no authorization rule, so every user who can log +in to the realm reaches the app +([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). + +Until that is fixed, check group membership in your app: verify the IdToken (see +Example 4 and [docs/auth-flow.md](docs/auth-flow.md)) and reject users whose +`groups` claim doesn't include the group you need. ## Troubleshooting diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 19001b4..5b21dd9 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -44,6 +44,7 @@ spec: - openid - profile - email + # groups is NOT enforced by the gateway in v0.1.1 - see spec.auth below groups: - admin gateway: public @@ -115,7 +116,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | | `clientSecretRef` | string | No | - | Name (string) of a Secret in the same namespace containing keys `client-id` and `client-secret`. **Note:** the spec field is a plain string (the Secret name), not the `{name, namespace}` object reference used by `status.clientSecretRef`. If omitted and `provisionClient` is true, the operator creates a Secret named `-oidc-client` with keys: `client-id`, `client-secret`, and `issuer-url`. | | `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | -| `groups` | []string | No | - | Groups that have access. When specified, only users in these groups are authorized. Case-sensitive. | +| `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](auth-flow.md#reading-user-identity-in-your-app)). | | `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | | `spaClient` | [SPAClientConfig](#specauthspaclient) | No | - | Provisions a public Keycloak client for browser-based PKCE flows (e.g., React apps using `keycloak-js`). Distinct from the confidential client used by gateway-enforced auth. | | `deviceFlowClient` | [DeviceFlowClientConfig](#specauthdeviceflowclient) | No | - | Provisions a public Keycloak client for the OAuth2 Device Authorization Grant (RFC 8628), for CLI/native apps. | @@ -305,10 +306,10 @@ spec: hostname: my-pack.nebari.example.com auth: enabled: true - groups: - - admin ``` +A strategic-merge patch only changes the fields it lists, so `routing` from the base is kept. + ### Helm Charts render the NebariApp through the shared `nebari-app.nebariApp` template diff --git a/docs/site/src/content/docs/concepts.md b/docs/site/src/content/docs/concepts.md index 02b1d24..6fd8c33 100644 --- a/docs/site/src/content/docs/concepts.md +++ b/docs/site/src/content/docs/concepts.md @@ -67,7 +67,7 @@ overlays/ nebariapp-patch.yaml # dev hostname, no auth production/ kustomization.yaml - nebariapp-patch.yaml # prod hostname, auth + groups + nebariapp-patch.yaml # prod hostname, auth enabled ``` Best for: packs deployed to multiple environments with known configuration differences. diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 376502a..90aeefe 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -46,6 +46,7 @@ spec: - openid - profile - email + # groups is NOT enforced by the gateway in v0.1.1 - see spec.auth below groups: - admin gateway: public @@ -117,7 +118,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | | `clientSecretRef` | string | No | - | Name (string) of a Secret in the same namespace containing keys `client-id` and `client-secret`. **Note:** the spec field is a plain string (the Secret name), not the `{name, namespace}` object reference used by `status.clientSecretRef`. If omitted and `provisionClient` is true, the operator creates a Secret named `-oidc-client` with keys: `client-id`, `client-secret`, and `issuer-url`. | | `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | -| `groups` | []string | No | - | Groups that have access. When specified, only users in these groups are authorized. Case-sensitive. | +| `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](/auth-flow/#reading-user-identity-in-your-app)). | | `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | | `spaClient` | [SPAClientConfig](#specauthspaclient) | No | - | Provisions a public Keycloak client for browser-based PKCE flows (e.g., React apps using `keycloak-js`). Distinct from the confidential client used by gateway-enforced auth. | | `deviceFlowClient` | [DeviceFlowClientConfig](#specauthdeviceflowclient) | No | - | Provisions a public Keycloak client for the OAuth2 Device Authorization Grant (RFC 8628), for CLI/native apps. | @@ -307,10 +308,10 @@ spec: hostname: my-pack.nebari.example.com auth: enabled: true - groups: - - admin ``` +A strategic-merge patch only changes the fields it lists, so `routing` from the base is kept. + ### Helm Charts render the NebariApp through the shared `nebari-app.nebariApp` template diff --git a/examples/kustomize-nginx/README.md b/examples/kustomize-nginx/README.md index 89f22cb..424c850 100644 --- a/examples/kustomize-nginx/README.md +++ b/examples/kustomize-nginx/README.md @@ -18,7 +18,7 @@ kustomize-nginx/ nebariapp-patch.yaml # Dev hostname, auth disabled production/ kustomization.yaml # Patches base for production - nebariapp-patch.yaml # Production hostname, auth enabled with groups + nebariapp-patch.yaml # Production hostname, auth enabled ``` ## When to use this approach @@ -76,7 +76,7 @@ spec: # Dev environment (auth disabled) kubectl apply -k examples/kustomize-nginx/overlays/dev/ -# Production environment (auth enabled, restricted to groups) +# Production environment (auth enabled) kubectl apply -k examples/kustomize-nginx/overlays/production/ ``` diff --git a/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml b/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml index 6bb075d..b095aaa 100644 --- a/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml +++ b/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml @@ -6,6 +6,8 @@ spec: hostname: my-pack.nebari.example.com auth: enabled: true + # Creates these groups in Keycloak. NOT enforced at the gateway in operator + # v0.1.1: any user in the realm can log in (nebari-operator#153). groups: - admin - data-science-team diff --git a/examples/vanilla-yaml/README.md b/examples/vanilla-yaml/README.md index 1ae7dea..f671dca 100644 --- a/examples/vanilla-yaml/README.md +++ b/examples/vanilla-yaml/README.md @@ -103,15 +103,11 @@ auth: enabled: true ``` -To restrict access to specific groups, add a `groups` list: - -```yaml -auth: - enabled: true - groups: - - admin - - data-science-team -``` +`auth.groups` does not restrict access in operator v0.1.1: every user who can log +in to the realm gets through +([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). +To limit access to a group, verify the IdToken in your app and check its `groups` +claim, as the auth-fastapi example does. ## Limitations From b8aa7685ecd382f77d485da165453922630c3ddf Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:17 +0200 Subject: [PATCH 06/16] Verify the IdToken in the auth example instead of trusting it (#37) The docs and auth-fastapi decoded the IdToken cookie without checking its signature, saying Envoy Gateway had verified it. Envoy's OAuth2 filter does not check JWT signatures, and requests can reach the app without passing it: any pod can call the Service, and publicRoutes have no SecurityPolicy. An unsigned token sent straight to the Service was shown as the logged-in user. - app: verify signature (JWKS), iss, aud and exp with PyJWT before using any claim; reject requests carrying more than one IdToken-* cookie; fail closed when verification is not configured - chart: pass client-id and issuer-url from the operator's OIDC Secret, with oidc.issuerURL and oidc.jwksURL overrides; add an optional NetworkPolicy that only admits the Envoy proxies (chart version bumped) - tests/: table-driven pytest suite, run in lint.yaml - integration CI: log in through Keycloak, and check forged tokens are rejected with and without the NetworkPolicy - Docs: auth-flow, README and the example README describe verification --- .github/workflows/lint.yaml | 18 +++ .github/workflows/test-integration.yaml | 26 +++- README.md | 59 +++++---- dev/Makefile | 8 +- docs/auth-flow.md | 71 ++++++---- docs/site/src/content/docs/auth-flow.md | 71 ++++++---- docs/site/src/content/docs/build-your-own.md | 2 +- docs/site/src/content/docs/concepts.md | 17 +-- docs/site/src/content/docs/index.md | 2 +- examples/auth-fastapi/README.md | 62 +++++++-- examples/auth-fastapi/app/main.py | 122 ++++++++++++------ examples/auth-fastapi/app/requirements.txt | 1 + .../auth-fastapi/app/templates/index.html | 8 +- examples/auth-fastapi/chart/Chart.yaml | 2 +- .../chart/templates/deployment.yaml | 23 ++++ .../chart/templates/networkpolicy.yaml | 33 +++++ examples/auth-fastapi/chart/values.yaml | 20 ++- examples/auth-fastapi/tests/conftest.py | 4 + examples/auth-fastapi/tests/requirements.txt | 2 + examples/auth-fastapi/tests/test_main.py | 99 ++++++++++++++ 20 files changed, 505 insertions(+), 145 deletions(-) create mode 100644 examples/auth-fastapi/chart/templates/networkpolicy.yaml create mode 100644 examples/auth-fastapi/tests/conftest.py create mode 100644 examples/auth-fastapi/tests/requirements.txt create mode 100644 examples/auth-fastapi/tests/test_main.py diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 4f9cede..2c45d46 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -77,6 +77,9 @@ jobs: - name: Template wrap-existing-chart (NebariApp enabled) run: helm template test examples/wrap-existing-chart/chart/ --set nebariapp.enabled=true --set nebariapp.hostname=test.example.com + - name: Template auth-fastapi (NetworkPolicy enabled) + run: helm template test examples/auth-fastapi/chart/ --set nebariapp.enabled=true --set nebariapp.hostname=test.example.com --set networkPolicy.enabled=true | kubeconform -strict -ignore-missing-schemas + # Without spec.routing the operator creates no HTTPRoute or TLS, yet the # NebariApp still reports Ready. Fail fast if an example chart drops it. - name: Every chart renders a NebariApp with routing @@ -94,3 +97,18 @@ jobs: if grep -rn --include='*.md' -E '^\s*project:\s*default\b' README.md docs examples; then echo "ArgoCD Application examples must use 'project: nebari-apps'"; exit 1 fi + + app-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-python@v6 + with: + python-version: "3.12" + + - name: Test auth-fastapi token verification + working-directory: examples/auth-fastapi + run: | + pip install -r app/requirements.txt -r tests/requirements.txt + pytest tests diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index d50180a..f51bf2d 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -180,7 +180,7 @@ jobs: run: helm uninstall test-basic --wait || true # ============================================================ - # Test: auth-fastapi (Helm, auth enabled) + # Test: auth-fastapi (Helm, auth enabled, full login) # ============================================================ - name: "Deploy auth-fastapi" run: | @@ -189,11 +189,33 @@ jobs: --set nebariapp.enabled=true \ --set nebariapp.hostname=fastapi.nebari.local \ --set nebariapp.auth.enabled=true \ - --wait --timeout 2m + --set oidc.jwksURL=http://keycloak-keycloakx-http.keycloak.svc.cluster.local/auth/realms/nebari/protocol/openid-connect/certs \ + --set networkPolicy.enabled=true \ + --wait --timeout 3m - name: "Verify auth-fastapi" run: | dev/verify-nebariapp.sh test-fastapi-my-pack fastapi.nebari.local --auth + # Log in through Keycloak and check the app shows a verified identity + dev/login-test.sh fastapi.nebari.local + + - name: "Verify auth-fastapi rejects forged tokens off the gateway" + run: | + # An unsigned token sent straight to the Service must not be trusted. + # With the NetworkPolicy on, the request should not connect at all; + # temporarily remove it to check the app's own verification as well. + FORGED='eyJhbGciOiJub25lIn0.eyJwcmVmZXJyZWRfdXNlcm5hbWUiOiJtYWxsb3J5In0.' + probe() { + kubectl run forged-probe --rm -i --restart=Never --quiet --image=curlimages/curl:8.11.1 -- \ + sh -c "curl -s -m 5 -H 'Cookie: IdToken-a1b2c3d4=$FORGED' http://test-fastapi-my-pack.default.svc.cluster.local/; echo curl_exit=\$?" + } + out=$(probe || true) + echo "$out" | grep -q 'curl_exit=28' || { echo "NetworkPolicy did not block direct access"; echo "$out"; exit 1; } + kubectl delete networkpolicy test-fastapi-my-pack --wait + out=$(probe || true) + echo "$out" | grep -q 'could not be verified' || { echo "forged token was not rejected"; echo "$out"; exit 1; } + if echo "$out" | grep -q mallory; then echo "forged identity was displayed"; exit 1; fi + echo "forged token rejected" - name: "Cleanup auth-fastapi" if: always() diff --git a/README.md b/README.md index 5dc2807..8100773 100644 --- a/README.md +++ b/README.md @@ -157,11 +157,13 @@ software-pack-template/ README.md auth-fastapi/ # Example 4: Custom app reading IdToken app/ - main.py # FastAPI reading IdToken cookie + main.py # FastAPI verifying the IdToken cookie requirements.txt templates/index.html # User info display + tests/ # pytest suite for the token verification Dockerfile - chart/ # Same structure as basic-nginx + chart/ # basic-nginx structure, plus OIDC env wiring + # and an optional networkpolicy.yaml README.md wrap-existing-chart/ # Example 5: Wrapping podinfo via Helm chart/ @@ -414,26 +416,27 @@ See [examples/basic-nginx/README.md](examples/basic-nginx/README.md) for the ful ## Example 4: Helm - Auth-Aware Pack (FastAPI) A custom Python app that reads the IdToken cookie set by Envoy Gateway after -Keycloak authentication. Shows how to consume authenticated user identity. +Keycloak authentication. Shows how to consume authenticated user identity safely. **What it demonstrates:** - Building a custom container image -- Reading the IdToken cookie to get user claims +- Verifying the IdToken's signature, issuer and audience against Keycloak before + trusting any claim (Envoy Gateway does not verify it, and requests can reach the + app without passing the gateway) +- Wiring the client ID and issuer from the operator's OIDC Secret in the chart +- An optional NetworkPolicy that only lets the Envoy proxies reach the pods - Rendering user info (username, email, groups) The key code in `app/main.py`: ```python -def get_id_token(request: Request) -> str | None: - """Extract IdToken from Envoy Gateway's OIDC filter cookies. - - Envoy Gateway sets a cookie named IdToken- where - is an 8-char hex string derived from the SecurityPolicy UID. - """ - for name, value in request.cookies.items(): - if name.startswith("IdToken-"): - return value - return None +claims = jwt.decode( + token, + jwks_client.get_signing_key_from_jwt(token).key, + algorithms=["RS256"], + audience=CLIENT_ID, # client-id from -oidc-client + issuer=ISSUER, # issuer-url from -oidc-client +) ``` ```bash @@ -444,7 +447,8 @@ docker run -p 8000:8000 ghcr.io/nebari-dev/software-pack-template/auth-fastapi-e helm dependency build examples/auth-fastapi/chart/ helm install my-pack examples/auth-fastapi/chart/ \ --set nebariapp.enabled=true \ - --set nebariapp.hostname=my-pack.nebari.example.com + --set nebariapp.hostname=my-pack.nebari.example.com \ + --set networkPolicy.enabled=true ``` See [examples/auth-fastapi/README.md](examples/auth-fastapi/README.md) for the full walkthrough. @@ -531,10 +535,18 @@ Gateway SecurityPolicy that handles the full OIDC flow: - Creates an HTTPRoute directing traffic to your service - Provisions a TLS certificate via cert-manager -**What your app can do:** -- Read the `IdToken-*` cookies to get the JWT (see Example 4) -- Decode the JWT payload to extract claims: `preferred_username`, `email`, `groups` -- The JWT signature is already verified by Envoy Gateway - you only need to base64-decode the payload +**What the operator does not do:** +- Enforce `auth.groups`. Any user who can log in to the realm reaches the app + ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). +- Verify the IdToken for your app. Envoy Gateway never checks the JWT signature, + and requests can reach the app without passing the gateway: from other pods, + and on `routing.publicRoutes` paths, which have no SecurityPolicy. + +**What your app must do before trusting an identity:** +- Read the single `IdToken-*` cookie (see Example 4) +- Verify its signature against Keycloak's JWKS, and check `iss`, `aud` and `exp` +- Only then use claims such as `preferred_username`, `email` and `groups`, for + example to restrict access to a group **If your app handles OAuth natively** (like Grafana), set `enforceAtGateway: false`. The operator will still provision the OIDC client and store credentials in a Secret, @@ -614,6 +626,7 @@ Runs on every push and PR. Validates all examples: - `helm lint` and `helm template` for each Helm chart (both NebariApp enabled and disabled) - Fails if any example chart renders a NebariApp without `spec.routing` - Fails if any ArgoCD Application example uses `project: default` (deny-all on NIC) +- Runs the auth-fastapi token verification tests (`pytest`) ### Build Images (`build-images.yaml`) @@ -656,9 +669,11 @@ infrastructure stack: `v0.1.0-alpha.20` (what NIC v0.14.0 deploys) - Configures the operator the way NIC does (`dev/configure-operator.sh`) - Deploys each example with NebariApp enabled and a `*.nebari.local` hostname -- Verifies NebariApp reaches `Ready` condition (HTTPRoute created, TLS configured) -- For auth-enabled examples (kustomize production, auth-fastapi), verifies - SecurityPolicy is created +- Verifies `RoutingReady` and `TLSReady`, that the HTTPRoute exists, and that a real + HTTPS request through the Gateway succeeds (`dev/verify-nebariapp.sh`) +- For auth-enabled examples, verifies `AuthReady`, the SecurityPolicy, and the + redirect to Keycloak; for auth-fastapi, logs in end to end and checks that forged + tokens sent around the gateway are rejected This catches bugs in NebariApp configuration, operator compatibility, and routing setup that the standalone test cannot detect. diff --git a/dev/Makefile b/dev/Makefile index 595b4c9..4043c91 100644 --- a/dev/Makefile +++ b/dev/Makefile @@ -26,6 +26,9 @@ HOSTNAME := my-pack.nebari.local CACHE_DIR := .cache # Ref-specific so changing OPERATOR_REF never reuses scripts cloned for another version. OPERATOR_REPO = $(CACHE_DIR)/nebari-operator-$(OPERATOR_REF) +# Pods cannot resolve keycloak.nebari.local, so apps that verify tokens fetch +# Keycloak's signing keys from the in-cluster Service instead. +KEYCLOAK_JWKS_URL := http://keycloak-keycloakx-http.keycloak.svc.cluster.local/auth/realms/nebari/protocol/openid-connect/certs # MetalLB IP pool configuration template. # METALLB_BASE_IP is substituted at runtime with the Kind network base. @@ -178,8 +181,9 @@ up-fastapi: cluster --set nebariapp.enabled=true \ --set nebariapp.hostname=$(HOSTNAME) \ --set nebariapp.auth.enabled=true \ - --wait --timeout 2m - kubectl wait --for=condition=Ready nebariapp/my-pack-my-pack --timeout=120s + --set oidc.jwksURL=$(KEYCLOAK_JWKS_URL) \ + --wait --timeout 3m + ./verify-nebariapp.sh my-pack $(HOSTNAME) --auth CLUSTER_NAME=$(CLUSTER_NAME) $(OPERATOR_REPO)/dev/scripts/networking/update-hosts.sh @echo "" @echo "FastAPI app deployed with NebariApp (auth enabled)." diff --git a/docs/auth-flow.md b/docs/auth-flow.md index e411451..a119812 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -120,35 +120,59 @@ For example: `IdToken-a1b2c3d4`, `AccessToken-a1b2c3d4`. Cookie names can be customized via the `cookieNames` field in the SecurityPolicy's OIDC configuration. -### Reading the IdToken in your app +### Reading user identity in your app -Find the cookie starting with `IdToken-`: +The IdToken is a JWT signed by Keycloak. **Verify its signature before you trust any claim +in it.** Envoy Gateway does not do this for you: -```python -for name, value in request.cookies.items(): - if name.startswith("IdToken-"): - full_token = value - break -``` +- Envoy's OAuth2 filter never checks the JWT signature. It reads the token only for its + expiry, and protects its own cookies with an HMAC. +- Requests can reach your app without passing through that filter. The Service is + reachable from any pod in the cluster unless you add a NetworkPolicy, and paths listed + in `routing.publicRoutes` are served by an HTTPRoute with no SecurityPolicy attached. + A request on any of those paths can carry any `IdToken-*` cookie it likes. -### Decoding the JWT payload +Verification needs three values, all available to your pod: -The IdToken is a standard JWT with three base64url-encoded sections separated by dots: -`header.payload.signature` +| Value | Where it comes from | +|-------|---------------------| +| Client ID (the token's `aud`) | `client-id` key of the `-oidc-client` Secret | +| Issuer (the token's `iss`) | `issuer-url` key of the same Secret. It is empty unless the operator runs with `KEYCLOAK_EXTERNAL_URL`; in that case use the issuer your Keycloak puts in tokens. | +| JWKS URL | Keycloak serves it at `/protocol/openid-connect/certs`. Use the in-cluster Keycloak URL if your pods cannot reach the public one. | -Since Envoy Gateway already verified the signature, you can safely decode just the -payload to extract claims: +With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10`): ```python -import base64, json - -parts = full_token.split(".") -payload = parts[1] -# Add base64 padding -payload += "=" * (4 - len(payload) % 4) -claims = json.loads(base64.urlsafe_b64decode(payload)) +import jwt + +jwks = jwt.PyJWKClient(JWKS_URL, cache_keys=True) + + +def verified_claims(request) -> dict | None: + tokens = [v for k, v in request.cookies.items() if k.startswith("IdToken-")] + if len(tokens) != 1: # none, or an extra cookie someone added + return None + try: + key = jwks.get_signing_key_from_jwt(tokens[0]) + return jwt.decode( + tokens[0], + key.key, + algorithms=["RS256"], + audience=CLIENT_ID, + issuer=ISSUER, + options={"require": ["exp", "iss", "aud"]}, + ) + except jwt.PyJWTError: + return None ``` +The [auth-fastapi example](https://github.com/nebari-dev/software-pack-template/tree/main/examples/auth-fastapi) +does this end to end, including wiring the three values from the Secret in its Helm chart. + +To keep traffic from bypassing the gateway entirely, also restrict ingress to your pods to +the Envoy proxies. The auth-fastapi chart ships an optional NetworkPolicy for this +(`networkPolicy.enabled: true`). + ### Common JWT claims | Claim | Description | @@ -196,8 +220,9 @@ metadata: data: client-id: # Always present. Value: - client-secret: # Always present. Cryptographically generated. - issuer-url: # Present when external consumers are configured. - # Value: Keycloak issuer URL (e.g., https://keycloak.example.com/realms/nebari) + issuer-url: # Always present. Empty unless the operator has + # KEYCLOAK_EXTERNAL_URL set; then the public issuer + # (e.g., https://keycloak.example.com/realms/nebari) spa-client-id: # Present when spaClient is enabled. device-client-id: # Present when deviceFlowClient is enabled. ``` @@ -278,7 +303,7 @@ with the OAuth flow, such as mapping Keycloak groups/roles to app-internal roles ### Gateway-only auth (app reads JWT from cookies) If your app just needs user identity (not role mapping), use `enforceAtGateway: true` -(the default) and read the IdToken cookie as described above. +(the default) and read and verify the IdToken cookie as described above. ### App-native auth only (no gateway enforcement) diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index b396f72..7ca9a8a 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -122,35 +122,59 @@ For example: `IdToken-a1b2c3d4`, `AccessToken-a1b2c3d4`. Cookie names can be customized via the `cookieNames` field in the SecurityPolicy's OIDC configuration. -### Reading the IdToken in your app +### Reading user identity in your app -Find the cookie starting with `IdToken-`: +The IdToken is a JWT signed by Keycloak. **Verify its signature before you trust any claim +in it.** Envoy Gateway does not do this for you: -```python -for name, value in request.cookies.items(): - if name.startswith("IdToken-"): - full_token = value - break -``` +- Envoy's OAuth2 filter never checks the JWT signature. It reads the token only for its + expiry, and protects its own cookies with an HMAC. +- Requests can reach your app without passing through that filter. The Service is + reachable from any pod in the cluster unless you add a NetworkPolicy, and paths listed + in `routing.publicRoutes` are served by an HTTPRoute with no SecurityPolicy attached. + A request on any of those paths can carry any `IdToken-*` cookie it likes. -### Decoding the JWT payload +Verification needs three values, all available to your pod: -The IdToken is a standard JWT with three base64url-encoded sections separated by dots: -`header.payload.signature` +| Value | Where it comes from | +|-------|---------------------| +| Client ID (the token's `aud`) | `client-id` key of the `-oidc-client` Secret | +| Issuer (the token's `iss`) | `issuer-url` key of the same Secret. It is empty unless the operator runs with `KEYCLOAK_EXTERNAL_URL`; in that case use the issuer your Keycloak puts in tokens. | +| JWKS URL | Keycloak serves it at `/protocol/openid-connect/certs`. Use the in-cluster Keycloak URL if your pods cannot reach the public one. | -Since Envoy Gateway already verified the signature, you can safely decode just the -payload to extract claims: +With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10`): ```python -import base64, json - -parts = full_token.split(".") -payload = parts[1] -# Add base64 padding -payload += "=" * (4 - len(payload) % 4) -claims = json.loads(base64.urlsafe_b64decode(payload)) +import jwt + +jwks = jwt.PyJWKClient(JWKS_URL, cache_keys=True) + + +def verified_claims(request) -> dict | None: + tokens = [v for k, v in request.cookies.items() if k.startswith("IdToken-")] + if len(tokens) != 1: # none, or an extra cookie someone added + return None + try: + key = jwks.get_signing_key_from_jwt(tokens[0]) + return jwt.decode( + tokens[0], + key.key, + algorithms=["RS256"], + audience=CLIENT_ID, + issuer=ISSUER, + options={"require": ["exp", "iss", "aud"]}, + ) + except jwt.PyJWTError: + return None ``` +The [auth-fastapi example](https://github.com/nebari-dev/software-pack-template/tree/main/examples/auth-fastapi) +does this end to end, including wiring the three values from the Secret in its Helm chart. + +To keep traffic from bypassing the gateway entirely, also restrict ingress to your pods to +the Envoy proxies. The auth-fastapi chart ships an optional NetworkPolicy for this +(`networkPolicy.enabled: true`). + ### Common JWT claims | Claim | Description | @@ -198,8 +222,9 @@ metadata: data: client-id: # Always present. Value: - client-secret: # Always present. Cryptographically generated. - issuer-url: # Present when external consumers are configured. - # Value: Keycloak issuer URL (e.g., https://keycloak.example.com/realms/nebari) + issuer-url: # Always present. Empty unless the operator has + # KEYCLOAK_EXTERNAL_URL set; then the public issuer + # (e.g., https://keycloak.example.com/realms/nebari) spa-client-id: # Present when spaClient is enabled. device-client-id: # Present when deviceFlowClient is enabled. ``` @@ -280,7 +305,7 @@ with the OAuth flow, such as mapping Keycloak groups/roles to app-internal roles ### Gateway-only auth (app reads JWT from cookies) If your app just needs user identity (not role mapping), use `enforceAtGateway: true` -(the default) and read the IdToken cookie as described above. +(the default) and read and verify the IdToken cookie as described above. ### App-native auth only (no gateway enforcement) diff --git a/docs/site/src/content/docs/build-your-own.md b/docs/site/src/content/docs/build-your-own.md index 9864d6d..7eea9fc 100644 --- a/docs/site/src/content/docs/build-your-own.md +++ b/docs/site/src/content/docs/build-your-own.md @@ -115,5 +115,5 @@ For a private OCI chart registry, use `type: helm`, `enableOCI: "true"`, and the ## Going deeper - [NebariApp CRD reference](/nebariapp-crd-reference/) - every field explained -- [Authentication flow](/auth-flow/) - how OIDC works end-to-end, including reading the IdToken in your app +- [Authentication flow](/auth-flow/) - how OIDC works end-to-end, including verifying the IdToken in your app - [Release readiness](/release-readiness/) - maturity levels and the promotion checklist for official packs diff --git a/docs/site/src/content/docs/concepts.md b/docs/site/src/content/docs/concepts.md index 6fd8c33..26fa876 100644 --- a/docs/site/src/content/docs/concepts.md +++ b/docs/site/src/content/docs/concepts.md @@ -133,18 +133,11 @@ The simplest possible Helm chart - nginx with a conditional NebariApp template a ### Example 4: Helm - Auth-Aware FastAPI A custom Python app that reads the `IdToken-*` cookie set by Envoy Gateway after -Keycloak authentication. The key snippet: - -```python -def get_id_token(request: Request) -> str | None: - for name, value in request.cookies.items(): - if name.startswith("IdToken-"): - return value - return None -``` - -Shows how to extract and decode the JWT to get `preferred_username`, `email`, and -`groups`. +Keycloak authentication, verifies the JWT's signature, issuer and audience against +Keycloak, and only then shows `preferred_username`, `email`, and `groups`. Envoy Gateway +does not verify the token for you, and requests can reach the app without passing the +gateway, so the verification is not optional. See +[Reading user identity in your app](/auth-flow/#reading-user-identity-in-your-app). ### Example 5: Helm - Wrapping an Existing Chart (Podinfo) diff --git a/docs/site/src/content/docs/index.md b/docs/site/src/content/docs/index.md index d99fc3c..618ea95 100644 --- a/docs/site/src/content/docs/index.md +++ b/docs/site/src/content/docs/index.md @@ -14,7 +14,7 @@ A software pack is a Kubernetes application bundled with a `NebariApp` custom re ## Reference pages - **[NebariApp CRD reference](/nebariapp-crd-reference/)** - complete field-by-field reference for the `NebariApp` custom resource -- **[Authentication flow](/auth-flow/)** - how OIDC works end-to-end; reading the IdToken in your app +- **[Authentication flow](/auth-flow/)** - how OIDC works end-to-end; verifying the IdToken in your app - **[Release readiness](/release-readiness/)** - maturity levels and the promotion checklist for first-party packs ## Pack template diff --git a/examples/auth-fastapi/README.md b/examples/auth-fastapi/README.md index dd17dd1..6163714 100644 --- a/examples/auth-fastapi/README.md +++ b/examples/auth-fastapi/README.md @@ -6,7 +6,10 @@ the IdToken cookie set by Envoy Gateway after Keycloak OIDC authentication. ## What This Example Shows - Building a custom container image for a Nebari pack -- Reading the IdToken cookie to extract user claims (username, email, groups) +- Verifying the IdToken cookie against Keycloak before trusting its claims + (username, email, groups) +- Wiring the operator's OIDC Secret into the app for that verification +- An optional NetworkPolicy so only the Envoy proxies can reach the pods - How Envoy Gateway handles the OIDC flow before requests reach your app - Full chart structure with Deployment, Service, and NebariApp @@ -20,10 +23,10 @@ When deployed on Nebari with auth enabled: 4. After login, Keycloak redirects back with an authorization code 5. Envoy Gateway exchanges the code for tokens and sets an `IdToken-*` cookie 6. The request (now with the cookie) is forwarded to the FastAPI app -7. The app decodes the JWT from the cookie and displays user info +7. The app verifies the JWT from the cookie against Keycloak and displays user info -Your app never handles the login flow - Envoy Gateway does it all. You just -read the resulting cookie. +Your app never handles the login flow - Envoy Gateway does it all. It does have +to verify the resulting cookie before trusting it (see the code walkthrough). ## Deploying to Nebari @@ -75,9 +78,24 @@ helm dependency build ./chart/ helm install my-pack ./chart/ \ --set nebariapp.enabled=true \ --set nebariapp.hostname=my-pack.nebari.example.com \ - --set nebariapp.auth.enabled=true + --set nebariapp.auth.enabled=true \ + --set networkPolicy.enabled=true ``` +The chart passes the app three values for token verification: + +| Env var | Source | Override | +|---------|--------|----------| +| `OIDC_CLIENT_ID` | `client-id` in the `-oidc-client` Secret | - | +| `OIDC_ISSUER_URL` | `issuer-url` in the same Secret | `oidc.issuerURL` | +| `OIDC_JWKS_URL` | `/protocol/openid-connect/certs` | `oidc.jwksURL` | + +The operator only fills `issuer-url` when it runs with `KEYCLOAK_EXTERNAL_URL` (NIC sets +it). If it is empty, set `oidc.issuerURL` to the issuer your Keycloak puts in tokens, +or the app shows every user as "Not Authenticated". Set `oidc.jwksURL` to the in-cluster +Keycloak URL if pods can't reach the public one. Env vars are read at startup, so restart +the pods after the Secret or these values change. + ## Local development The container image is pre-built and published to GHCR by CI. You can run it @@ -100,9 +118,17 @@ docker run -p 8000:8000 my-pack-fastapi:latest ### `app/main.py` -The key function is `get_id_token()` which extracts the JWT from Envoy Gateway's -`IdToken-` cookie. The JWT payload is base64-decoded (no signature -verification needed since Envoy Gateway already verified it) to extract claims: +`get_id_token()` returns the JWT from Envoy Gateway's `IdToken-` cookie, +and refuses the request if there is more than one such cookie. `TokenVerifier` +then checks the signature against Keycloak's JWKS, plus `iss`, `aud` and `exp`, +before any claim is used. + +Envoy Gateway does not verify the JWT signature itself; its OAuth2 filter only reads +the expiry and protects its own cookies with an HMAC. And requests can reach the app +without passing that filter: other pods can call the Service directly, and +`routing.publicRoutes` paths have no SecurityPolicy. So an app that decodes the cookie +without verifying it trusts whatever identity the caller sends. Once verified, the +app reads: - `preferred_username` - the Keycloak username - `email` - user's email address @@ -113,13 +139,26 @@ verification needed since Envoy Gateway already verified it) to extract claims: ### `app/templates/index.html` A simple Jinja2 template that renders user information when authenticated, or -an explanatory message when no IdToken is present. +an explanatory message saying why no identity is shown (no cookie, verification +failed, or verification not configured). + +### `tests/` + +Table-driven tests for the verification: valid token, wrong key, unknown key ID, wrong +audience or issuer, expired, unsigned, and an extra `IdToken-*` cookie. + +```bash +cd examples/auth-fastapi +pip install -r app/requirements.txt -r tests/requirements.txt +pytest tests +``` ## Files | File | Purpose | |------|---------| -| `app/main.py` | FastAPI application reading IdToken cookies | +| `app/main.py` | FastAPI application verifying IdToken cookies | +| `tests/test_main.py` | Tests for the token verification | | `app/requirements.txt` | Python dependencies | | `app/templates/index.html` | HTML template for user info display | | `Dockerfile` | Multi-stage build for the FastAPI image | @@ -127,7 +166,8 @@ an explanatory message when no IdToken is present. | `chart/values.yaml` | Default config, including the NebariApp spec with auth enabled | | `chart/templates/_helpers.tpl` | Name, label, and selector helpers | | `chart/templates/nebariapp.yaml` | Renders the NebariApp via nebari-app | -| `chart/templates/deployment.yaml` | Kubernetes Deployment | +| `chart/templates/deployment.yaml` | Kubernetes Deployment, with the OIDC env vars when auth is on | +| `chart/templates/networkpolicy.yaml` | Optional NetworkPolicy (`networkPolicy.enabled`) | | `chart/templates/service.yaml` | ClusterIP Service | | `chart/templates/NOTES.txt` | Post-install instructions | diff --git a/examples/auth-fastapi/app/main.py b/examples/auth-fastapi/app/main.py index d95639e..1077116 100644 --- a/examples/auth-fastapi/app/main.py +++ b/examples/auth-fastapi/app/main.py @@ -6,61 +6,90 @@ When deployed on Nebari with auth enabled, the Envoy Gateway OIDC filter handles the login flow and sets an IdToken cookie containing the JWT. -This app reads that cookie to display user information. +This app verifies that JWT against Keycloak's signing keys before trusting +any claim in it. """ -import base64 -import json +import logging import os from pathlib import Path +import jwt from fastapi import FastAPI, Request from fastapi.responses import HTMLResponse from fastapi.templating import Jinja2Templates +log = logging.getLogger(__name__) + app = FastAPI(title="Auth-Aware Nebari Pack") templates = Jinja2Templates(directory=Path(__file__).parent / "templates") -def decode_jwt_payload(token: str) -> dict: - """Decode the payload section of a JWT without verifying the signature. +class TokenVerifier: + """Verifies IdTokens issued by the Keycloak realm for this app's OIDC client. - This is safe because the token was already verified by Envoy Gateway. - We only need to extract the claims for display purposes. + Envoy Gateway does NOT verify the JWT signature: its OAuth2 filter only + reads the token's expiry and protects its own cookies with an HMAC. Requests + can also reach the app without passing that filter at all (in-cluster + traffic to the Service, or paths in routing.publicRoutes), carrying any + cookie the caller likes. So the app has to check the signature, issuer, + audience and expiry itself. """ - parts = token.split(".") - if len(parts) != 3: - return {} - # JWT payload is base64url-encoded - payload = parts[1] - # Add padding if needed - padding = 4 - len(payload) % 4 - if padding != 4: - payload += "=" * padding + def __init__(self, issuer: str, client_id: str, jwks_client): + self.issuer = issuer + self.client_id = client_id + self.jwks_client = jwks_client + + def verify(self, token: str) -> dict | None: + try: + key = self.jwks_client.get_signing_key_from_jwt(token) + return jwt.decode( + token, + key.key, + algorithms=["RS256"], + audience=self.client_id, + issuer=self.issuer, + # PyJWT rejects a missing iss/aud when they are passed above, + # but accepts a token with no exp unless told to require it. + options={"require": ["exp", "iss", "aud"]}, + ) + except jwt.PyJWTError as exc: + log.info("rejected IdToken: %s", exc) + return None + + +def verifier_from_env() -> TokenVerifier | None: + """Build a verifier from the values the Helm chart injects. + + OIDC_CLIENT_ID and OIDC_ISSUER_URL come from the operator-created + -oidc-client Secret (keys client-id and issuer-url). + OIDC_JWKS_URL defaults to Keycloak's certs endpoint under the issuer; set it + to the in-cluster Keycloak URL when pods cannot reach the public one. + """ + issuer = os.environ.get("OIDC_ISSUER_URL", "") + client_id = os.environ.get("OIDC_CLIENT_ID", "") + if not issuer or not client_id: + return None + jwks_url = os.environ.get("OIDC_JWKS_URL") or f"{issuer.rstrip('/')}/protocol/openid-connect/certs" + return TokenVerifier(issuer, client_id, jwt.PyJWKClient(jwks_url, cache_keys=True, timeout=5)) - try: - decoded = base64.urlsafe_b64decode(payload) - return json.loads(decoded) - except (ValueError, json.JSONDecodeError): - return {} +verifier = verifier_from_env() -def get_id_token(request: Request) -> str | None: - """Extract the IdToken from Envoy Gateway's OIDC filter cookies. - Envoy Gateway's OIDC filter sets a cookie named IdToken- where - is an 8-character hex string derived from the SecurityPolicy UID. - For example: IdToken-a1b2c3d4 +def get_id_token(request: Request) -> str | None: + """Return the IdToken cookie set by Envoy Gateway's OIDC filter. - We look for any cookie starting with "IdToken-" to avoid hardcoding - the suffix value. + Envoy Gateway names the cookie IdToken-, where is generated + per SecurityPolicy, so the app matches on the prefix. Exactly one such + cookie is expected; if there are more, someone added one, and the request + is treated as unauthenticated rather than guessing which to trust. """ - for name, value in request.cookies.items(): - if name.startswith("IdToken-"): - return value - - return None + tokens = [value for name, value in request.cookies.items() if name.startswith("IdToken-")] + if len(tokens) != 1: + return None + return tokens[0] @app.get("/health") @@ -74,16 +103,22 @@ def index(request: Request): """Main page showing authenticated user information.""" token = get_id_token(request) user_info = None - - if token: - claims = decode_jwt_payload(token) - user_info = { - "username": claims.get("preferred_username", "unknown"), - "email": claims.get("email", ""), - "name": claims.get("name", ""), - "groups": claims.get("groups", []), - "roles": claims.get("realm_access", {}).get("roles", []), - } + reason = "No IdToken cookie found." + + if verifier is None: + reason = "Token verification is not configured (OIDC_ISSUER_URL and OIDC_CLIENT_ID are unset)." + elif token: + claims = verifier.verify(token) + if claims is None: + reason = "The IdToken cookie could not be verified." + else: + user_info = { + "username": claims.get("preferred_username", "unknown"), + "email": claims.get("email", ""), + "name": claims.get("name", ""), + "groups": claims.get("groups", []), + "roles": claims.get("realm_access", {}).get("roles", []), + } return templates.TemplateResponse( "index.html", @@ -91,6 +126,7 @@ def index(request: Request): "request": request, "user_info": user_info, "authenticated": user_info is not None, + "reason": reason, }, ) diff --git a/examples/auth-fastapi/app/requirements.txt b/examples/auth-fastapi/app/requirements.txt index 6a07fdd..e128b73 100644 --- a/examples/auth-fastapi/app/requirements.txt +++ b/examples/auth-fastapi/app/requirements.txt @@ -1,3 +1,4 @@ fastapi==0.115.8 uvicorn[standard]==0.34.0 jinja2==3.1.5 +PyJWT[crypto]==2.15.1 diff --git a/examples/auth-fastapi/app/templates/index.html b/examples/auth-fastapi/app/templates/index.html index a61d046..42b23a7 100644 --- a/examples/auth-fastapi/app/templates/index.html +++ b/examples/auth-fastapi/app/templates/index.html @@ -75,14 +75,16 @@

Authenticated User

This information was extracted from the IdToken cookie - set by Envoy Gateway after Keycloak authentication. + set by Envoy Gateway after Keycloak authentication, after verifying + its signature against Keycloak's signing keys.

{% else %}

Not Authenticated

- No IdToken cookie found. This app reads authenticated user identity - from the IdToken cookie set by Envoy Gateway's OIDC filter. + {{ reason }} This app reads authenticated user identity from the + IdToken cookie set by Envoy Gateway's OIDC filter, and only trusts it + once the signature checks out.

To see authentication in action:

    diff --git a/examples/auth-fastapi/chart/Chart.yaml b/examples/auth-fastapi/chart/Chart.yaml index 1d72e4c..f4a8a72 100644 --- a/examples/auth-fastapi/chart/Chart.yaml +++ b/examples/auth-fastapi/chart/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: my-pack description: A Nebari Software Pack with auth-aware FastAPI application type: application -version: 0.1.2 +version: 0.1.3 appVersion: "1.0.0" dependencies: - name: nebari-app diff --git a/examples/auth-fastapi/chart/templates/deployment.yaml b/examples/auth-fastapi/chart/templates/deployment.yaml index 817be70..275b3b5 100644 --- a/examples/auth-fastapi/chart/templates/deployment.yaml +++ b/examples/auth-fastapi/chart/templates/deployment.yaml @@ -23,6 +23,29 @@ spec: - name: http containerPort: {{ .Values.service.targetPort }} protocol: TCP + {{- if and .Values.nebariapp.enabled .Values.nebariapp.auth.enabled }} + # The app verifies the IdToken itself; Envoy Gateway does not check its + # signature. Client ID and issuer come from the operator's OIDC Secret. + env: + - name: OIDC_CLIENT_ID + valueFrom: + secretKeyRef: + name: {{ include "my-pack.fullname" . }}-oidc-client + key: client-id + - name: OIDC_ISSUER_URL + {{- if .Values.oidc.issuerURL }} + value: {{ .Values.oidc.issuerURL | quote }} + {{- else }} + valueFrom: + secretKeyRef: + name: {{ include "my-pack.fullname" . }}-oidc-client + key: issuer-url + {{- end }} + {{- with .Values.oidc.jwksURL }} + - name: OIDC_JWKS_URL + value: {{ . | quote }} + {{- end }} + {{- end }} livenessProbe: httpGet: path: /health diff --git a/examples/auth-fastapi/chart/templates/networkpolicy.yaml b/examples/auth-fastapi/chart/templates/networkpolicy.yaml new file mode 100644 index 0000000..25f8499 --- /dev/null +++ b/examples/auth-fastapi/chart/templates/networkpolicy.yaml @@ -0,0 +1,33 @@ +{{/* + Optional: only let the Envoy Gateway proxies reach the app's pods, so requests + cannot bypass the OIDC filter by calling the Service from inside the cluster. + This does not cover routing.publicRoutes, which go through the gateway without + a SecurityPolicy; the app's own token verification handles those. +*/}} +{{- if .Values.networkPolicy.enabled }} +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "my-pack.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "my-pack.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + {{- include "my-pack.selectorLabels" . | nindent 6 }} + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: {{ .Values.networkPolicy.gatewayNamespace }} + podSelector: + matchLabels: + app.kubernetes.io/name: envoy + app.kubernetes.io/component: proxy + ports: + - port: {{ .Values.service.targetPort }} + protocol: TCP +{{- end }} diff --git a/examples/auth-fastapi/chart/values.yaml b/examples/auth-fastapi/chart/values.yaml index e9e701a..3d0112e 100644 --- a/examples/auth-fastapi/chart/values.yaml +++ b/examples/auth-fastapi/chart/values.yaml @@ -28,7 +28,7 @@ resources: {} # Nebari Integration # ============================================================================= # NebariApp CR (https://github.com/nebari-dev/nebari-operator). -# Rendered through the official nebari-app.nebariApp template +# Rendered through the official nebari-app.nebariApp template # (https://github.com/nebari-dev/nebari-operator/tree/main/charts/nebari-app) with templating enabled. nebariapp: enabled: false @@ -56,6 +56,24 @@ nebariapp: - email gateway: public +# ============================================================================= +# Token verification +# ============================================================================= +# The app verifies the IdToken signature itself, because Envoy Gateway does not. +# With nebariapp.auth.enabled, the client ID and issuer are read from the +# operator-created -oidc-client Secret. Override them when that +# Secret's issuer-url is empty (operator running without KEYCLOAK_EXTERNAL_URL) +# or when pods cannot reach the public Keycloak URL. +oidc: + issuerURL: "" # expected `iss` claim; defaults to the Secret's issuer-url + jwksURL: "" # defaults to /protocol/openid-connect/certs + +# Only allow the Envoy Gateway proxies to reach the app's pods. Requires a CNI +# that enforces NetworkPolicy. +networkPolicy: + enabled: false + gatewayNamespace: envoy-gateway-system + # ============================================================================= # Overrides # ============================================================================= diff --git a/examples/auth-fastapi/tests/conftest.py b/examples/auth-fastapi/tests/conftest.py new file mode 100644 index 0000000..605c442 --- /dev/null +++ b/examples/auth-fastapi/tests/conftest.py @@ -0,0 +1,4 @@ +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "app")) diff --git a/examples/auth-fastapi/tests/requirements.txt b/examples/auth-fastapi/tests/requirements.txt new file mode 100644 index 0000000..7e9e6d9 --- /dev/null +++ b/examples/auth-fastapi/tests/requirements.txt @@ -0,0 +1,2 @@ +pytest==8.4.2 +httpx==0.28.1 diff --git a/examples/auth-fastapi/tests/test_main.py b/examples/auth-fastapi/tests/test_main.py new file mode 100644 index 0000000..c2b3e8b --- /dev/null +++ b/examples/auth-fastapi/tests/test_main.py @@ -0,0 +1,99 @@ +"""Tests for IdToken verification in the auth-fastapi example. + +Run from examples/auth-fastapi: + pip install -r app/requirements.txt -r tests/requirements.txt + pytest tests +""" + +import time + +import jwt +import pytest +from cryptography.hazmat.primitives.asymmetric import rsa +from fastapi.testclient import TestClient + +import main + +ISSUER = "https://keycloak.example.com/realms/nebari" +CLIENT_ID = "default-my-pack" + + +def _keypair(): + return rsa.generate_private_key(public_exponent=65537, key_size=2048) + + +TRUSTED = _keypair() +UNTRUSTED = _keypair() + + +class FakeJWKClient: + """Stands in for jwt.PyJWKClient: serves only the trusted public key.""" + + def get_signing_key_from_jwt(self, token): + header = jwt.get_unverified_header(token) + if header.get("kid") != "trusted": + raise jwt.PyJWKClientError("unknown kid") + return jwt.PyJWK.from_dict( + {**jwt.algorithms.RSAAlgorithm.to_jwk(TRUSTED.public_key(), as_dict=True), "kid": "trusted"} + ) + + +def make_token(key=TRUSTED, kid="trusted", drop=(), **overrides): + claims = { + "iss": ISSUER, + "aud": CLIENT_ID, + "exp": int(time.time()) + 300, + "preferred_username": "alice", + "email": "alice@example.com", + "groups": ["admin"], + } + claims.update(overrides) + for name in drop: + del claims[name] + return jwt.encode(claims, key, algorithm="RS256", headers={"kid": kid}) + + +@pytest.fixture +def client(monkeypatch): + monkeypatch.setattr(main, "verifier", main.TokenVerifier(ISSUER, CLIENT_ID, FakeJWKClient())) + return TestClient(main.app) + + +@pytest.mark.parametrize( + "name, cookies, authenticated", + [ + ("valid token", {"IdToken-a1b2c3d4": make_token()}, True), + ("no cookie", {}, False), + ("signed by another key", {"IdToken-a1b2c3d4": make_token(key=UNTRUSTED)}, False), + ("unknown kid", {"IdToken-a1b2c3d4": make_token(kid="other")}, False), + ("wrong audience", {"IdToken-a1b2c3d4": make_token(aud="someone-else")}, False), + ("wrong issuer", {"IdToken-a1b2c3d4": make_token(iss="https://evil.example.com")}, False), + ("expired", {"IdToken-a1b2c3d4": make_token(exp=int(time.time()) - 60)}, False), + ("no expiry", {"IdToken-a1b2c3d4": make_token(drop=["exp"])}, False), + ("unsigned payload", {"IdToken-a1b2c3d4": "e30.eyJwcmVmZXJyZWRfdXNlcm5hbWUiOiJldmUifQ."}, False), + ( + "extra IdToken cookie", + {"IdToken-a1b2c3d4": make_token(), "IdToken-00000000": make_token(key=UNTRUSTED)}, + False, + ), + ], +) +def test_index_only_trusts_verified_tokens(client, name, cookies, authenticated): + for k, v in cookies.items(): + client.cookies.set(k, v) + body = client.get("/").text + assert ("alice" in body) is authenticated, name + assert ("Not Authenticated" in body) is not authenticated, name + + +def test_verification_not_configured_fails_closed(monkeypatch): + monkeypatch.setattr(main, "verifier", None) + client = TestClient(main.app) + client.cookies.set("IdToken-a1b2c3d4", make_token()) + body = client.get("/").text + assert "alice" not in body + assert "not configured" in body + + +def test_health(): + assert TestClient(main.app).get("/health").json() == {"status": "ok"} From 4d0319dbd7c5b0f7db87644b48bcc57e4877c06d Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:18 +0200 Subject: [PATCH 07/16] Correct resource names and locations in the auth and troubleshooting docs (#38) - Auth Flow: SecurityPolicy is -security targeting -route, HTTPRoutes are -route and -public-route, and Certificates are --cert in envoy-gateway-system; show the endpoints and logoutPath the operator actually sets - README troubleshooting: look for Certificates in envoy-gateway-system, the operator in nebari-operator-system, and Envoy Gateway logs via its Deployment --- README.md | 15 ++++++++----- docs/auth-flow.md | 29 +++++++++++++++++++------ docs/site/src/content/docs/auth-flow.md | 29 +++++++++++++++++++------ 3 files changed, 53 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 8100773..5fa56f2 100644 --- a/README.md +++ b/README.md @@ -938,7 +938,7 @@ naming convention (usually `-`). 1. Check that `auth.enabled` is `true` in the NebariApp spec 2. Check that the nebari-operator is running: ```bash - kubectl get pods -n nebari-system -l app=nebari-operator + kubectl get pods -n nebari-operator-system ``` 3. Check the NebariApp conditions: ```bash @@ -962,14 +962,17 @@ Service (NIC sets these; `dev/configure-operator.sh` shows the kind equivalent). ### TLS certificate not provisioning -1. Check cert-manager is running: +1. Check the `TLSReady` reason. `ClusterIssuerNotConfigured` means the operator has + no `TLS_CLUSTER_ISSUER_NAME`; the app is then served by the Gateway's shared + listener and certificate. +2. Check cert-manager is running: ```bash kubectl get pods -n cert-manager ``` -2. Check the Certificate resource: +3. Check the Certificate resource. Certificates live in the Gateway's namespace: ```bash - kubectl get certificate -n my-pack - kubectl describe certificate my-pack-tls -n my-pack + kubectl get certificate -n envoy-gateway-system + kubectl describe certificate my-pack-my-pack-cert -n envoy-gateway-system ``` ### No IdToken cookie in the app @@ -981,7 +984,7 @@ Service (NIC sets these; `dev/configure-operator.sh` shows the kind equivalent). ``` 3. Check Envoy Gateway logs: ```bash - kubectl logs -n envoy-gateway-system -l app=envoy-gateway + kubectl logs -n envoy-gateway-system deploy/envoy-gateway ``` ### `missing in charts/ directory: nebari-app` diff --git a/docs/auth-flow.md b/docs/auth-flow.md index a119812..17a26c4 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -242,29 +242,39 @@ without additional RBAC configuration. apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: - name: -oidc + name: -security spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute - name: + name: -route oidc: provider: - issuer: https:///realms/ - clientID: + # In-cluster Keycloak URL, used only by Envoy Gateway's control plane + issuer: http://..svc.cluster.local/.../realms/ + tokenEndpoint: /protocol/openid-connect/token + # Browser-facing endpoints, set when the operator has KEYCLOAK_EXTERNAL_URL + authorizationEndpoint: /protocol/openid-connect/auth + endSessionEndpoint: /protocol/openid-connect/logout + clientID: - clientSecret: name: -oidc-client redirectURL: https:// + logoutPath: /logout scopes: [openid, profile, email] ``` +The policy targets only `-route`. Paths in `routing.publicRoutes` are +served by a second HTTPRoute, `-public-route`, which has no SecurityPolicy. +The policy contains no authorization rules: `auth.groups` is not enforced here. + ### 4. HTTPRoute ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: - name: + name: -route # plus -public-route for publicRoutes spec: parentRefs: - name: @@ -279,13 +289,18 @@ spec: ### 5. cert-manager Certificate (when `routing.tls.enabled: true`) +Created only when the operator has a cert-manager ClusterIssuer configured and +`routing.tls.secretName` is not set. Certificates live in the Gateway's namespace, so +their names include the app's namespace: + ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: - name: -tls + name: --cert + namespace: envoy-gateway-system spec: - secretName: -tls + secretName: --tls dnsNames: - issuerRef: diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index 7ca9a8a..eabe89f 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -244,29 +244,39 @@ without additional RBAC configuration. apiVersion: gateway.envoyproxy.io/v1alpha1 kind: SecurityPolicy metadata: - name: -oidc + name: -security spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute - name: + name: -route oidc: provider: - issuer: https:///realms/ - clientID: + # In-cluster Keycloak URL, used only by Envoy Gateway's control plane + issuer: http://..svc.cluster.local/.../realms/ + tokenEndpoint: /protocol/openid-connect/token + # Browser-facing endpoints, set when the operator has KEYCLOAK_EXTERNAL_URL + authorizationEndpoint: /protocol/openid-connect/auth + endSessionEndpoint: /protocol/openid-connect/logout + clientID: - clientSecret: name: -oidc-client redirectURL: https:// + logoutPath: /logout scopes: [openid, profile, email] ``` +The policy targets only `-route`. Paths in `routing.publicRoutes` are +served by a second HTTPRoute, `-public-route`, which has no SecurityPolicy. +The policy contains no authorization rules: `auth.groups` is not enforced here. + ### 4. HTTPRoute ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: - name: + name: -route # plus -public-route for publicRoutes spec: parentRefs: - name: @@ -281,13 +291,18 @@ spec: ### 5. cert-manager Certificate (when `routing.tls.enabled: true`) +Created only when the operator has a cert-manager ClusterIssuer configured and +`routing.tls.secretName` is not set. Certificates live in the Gateway's namespace, so +their names include the app's namespace: + ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: - name: -tls + name: --cert + namespace: envoy-gateway-system spec: - secretName: -tls + secretName: --tls dnsNames: - issuerRef: From bf575206c7026ec854fba3020e555979cd3ae3aa Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:18 +0200 Subject: [PATCH 08/16] Describe what the OIDC Secret Role does and does not restrict (#39) The CRD reference said only the app's pods can read the OIDC client Secret. The operator's Role grants one ServiceAccount API access; RBAC is additive, and any pod in the namespace can mount the Secret through the kubelet. - CRD reference: reword serviceAccountName and add "Who can read the OIDC Secret" - Auth Flow: secretKeyRef works without the Role; the Role only matters for API reads --- docs/auth-flow.md | 10 ++++++---- docs/nebariapp-crd-reference.md | 17 ++++++++++++++++- docs/site/src/content/docs/auth-flow.md | 10 ++++++---- .../content/docs/nebariapp-crd-reference.md | 18 +++++++++++++++++- 4 files changed, 45 insertions(+), 10 deletions(-) diff --git a/docs/auth-flow.md b/docs/auth-flow.md index 17a26c4..18e1291 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -227,14 +227,16 @@ data: device-client-id: # Present when deviceFlowClient is enabled. ``` -The operator also creates RBAC resources granting your app's ServiceAccount read -access to the secret: +The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` `get` +this Secret through the Kubernetes API: - **Role:** `-oidc-secret-reader` - **RoleBinding:** `-oidc-secret-reader` -This means your app's pods can reference the secret in `env.valueFrom.secretKeyRef` -without additional RBAC configuration. +You only need this if your app reads the Secret through the API. Referencing it with +`env.valueFrom.secretKeyRef` or a volume works without any Role, because the kubelet +fetches it. The Role does not stop anyone else from reading the Secret either; see +"Who can read the OIDC Secret" in the NebariApp CRD reference. ### 3. Envoy Gateway SecurityPolicy (when `enforceAtGateway: true`) diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 5b21dd9..7503a3e 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -66,7 +66,7 @@ spec: | `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | | `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | | `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | -| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. The operator scopes RBAC on the OIDC client Secret to this ServiceAccount, so only the app's pods can read its credentials. | +| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. When the operator provisions an OIDC client, it creates a Role and RoleBinding that let this ServiceAccount `get` the OIDC client Secret through the Kubernetes API. This grants access; it does not restrict anyone else. See [who can read the OIDC Secret](#who-can-read-the-oidc-secret). | | `landingPage` | [LandingPageConfig](#speclandingpage) | No | - | Controls how this service appears on the Nebari landing page. | ## spec.service @@ -262,6 +262,21 @@ spec: ArgoCD only applies `managedNamespaceMetadata` to a namespace that the same Application creates. If the namespace already exists, label it yourself. +## Who can read the OIDC Secret + +When `provisionClient` is true, the operator writes `-oidc-client` and creates a Role +(`-oidc-secret-reader`) that lets `spec.serviceAccountName` `get` that Secret through the +Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from +anyone else. Kubernetes RBAC is additive, so these can also read the Secret: + +- anyone who already has `get` on Secrets in the namespace, such as namespace admins +- any pod in the namespace, under any ServiceAccount, that mounts the Secret as an env var or + volume (the kubelet fetches it, not the pod's ServiceAccount), and therefore anyone who can + create pods there +- the operator, which has cluster-wide Secret access, and Envoy Gateway, which reads the + client secret to run the OIDC filter + +Treat the namespace as the security boundary for these credentials. ## Deployment Patterns diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index eabe89f..0c0662c 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -229,14 +229,16 @@ data: device-client-id: # Present when deviceFlowClient is enabled. ``` -The operator also creates RBAC resources granting your app's ServiceAccount read -access to the secret: +The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` `get` +this Secret through the Kubernetes API: - **Role:** `-oidc-secret-reader` - **RoleBinding:** `-oidc-secret-reader` -This means your app's pods can reference the secret in `env.valueFrom.secretKeyRef` -without additional RBAC configuration. +You only need this if your app reads the Secret through the API. Referencing it with +`env.valueFrom.secretKeyRef` or a volume works without any Role, because the kubelet +fetches it. The Role does not stop anyone else from reading the Secret either; see +"Who can read the OIDC Secret" in the NebariApp CRD reference. ### 3. Envoy Gateway SecurityPolicy (when `enforceAtGateway: true`) diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 90aeefe..72e1d08 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -68,7 +68,7 @@ spec: | `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | | `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | | `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | -| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. The operator scopes RBAC on the OIDC client Secret to this ServiceAccount, so only the app's pods can read its credentials. | +| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. When the operator provisions an OIDC client, it creates a Role and RoleBinding that let this ServiceAccount `get` the OIDC client Secret through the Kubernetes API. This grants access; it does not restrict anyone else. See [who can read the OIDC Secret](#who-can-read-the-oidc-secret). | | `landingPage` | [LandingPageConfig](#speclandingpage) | No | - | Controls how this service appears on the Nebari landing page. | ## spec.service @@ -265,6 +265,22 @@ spec: ArgoCD only applies `managedNamespaceMetadata` to a namespace that the same Application creates. If the namespace already exists, label it yourself. +## Who can read the OIDC Secret + +When `provisionClient` is true, the operator writes `-oidc-client` and creates a Role +(`-oidc-secret-reader`) that lets `spec.serviceAccountName` `get` that Secret through the +Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from +anyone else. Kubernetes RBAC is additive, so these can also read the Secret: + +- anyone who already has `get` on Secrets in the namespace, such as namespace admins +- any pod in the namespace, under any ServiceAccount, that mounts the Secret as an env var or + volume (the kubelet fetches it, not the pod's ServiceAccount), and therefore anyone who can + create pods there +- the operator, which has cluster-wide Secret access, and Envoy Gateway, which reads the + client secret to run the OIDC filter + +Treat the namespace as the security boundary for these credentials. + ## Deployment Patterns The NebariApp resource can be included in your pack using any deployment method. From 61937c19d6b5f76cd669274117128a17b1a98f8b Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 13:47:19 +0200 Subject: [PATCH 09/16] Bring the CRD reference up to date with operator v0.1.1 (#41) - Add routing.tls.secretName and landingPage.iconLight/iconDark, marked with the release that introduced them - Replace the condition reason table with every reason the v0.1.1 controllers set, grouped by condition, and note the defined-but-unused ones - Explain that Ready does not wait for RoutingReady, TLSReady or AuthReady - Mark clientSecretRef, status.gatewayRef and status.clientSecretRef as not implemented, and fix the displayName and externalUrl descriptions - Replace the stale hand-written Helm template with the nebari-app library chart pattern - README: explain TLSReady=False/ClusterIssuerNotConfigured --- README.md | 4 +- docs/nebariapp-crd-reference.md | 71 ++++++++++++------- .../content/docs/nebariapp-crd-reference.md | 71 ++++++++++++------- 3 files changed, 93 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index 5fa56f2..aa87c08 100644 --- a/README.md +++ b/README.md @@ -836,7 +836,9 @@ kubectl describe nebariapp my-pack -n my-pack # Expected conditions: # RoutingReady: True - HTTPRoute created -# TLSReady: True - Certificate provisioned +# TLSReady: True - Certificate provisioned (False/ClusterIssuerNotConfigured +# if the operator has no ClusterIssuer; the shared +# wildcard listener still serves HTTPS) # AuthReady: True - SecurityPolicy created (if auth enabled) # Ready: True - Core checks passed. Does NOT wait for the others: # a NebariApp with no routing is Ready and unreachable. diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 7503a3e..286adf7 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -100,6 +100,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `enabled` | *bool | No | `true` | Whether to provision a TLS certificate via cert-manager and configure an HTTPS listener on the Gateway. When `false`, only HTTP listeners are used. | +| `secretName` | string | No | - | *(v0.1.0-alpha.20+)* Name of a pre-existing `kubernetes.io/tls` Secret in `envoy-gateway-system` to use for the HTTPS listener instead of a cert-manager Certificate (for wildcard or externally managed certs). When set, the operator creates no Certificate and cleans up any it owns. You create and rotate the Secret. Ignored when `enabled` is `false`. Max 253 chars, DNS-subdomain format. | ## spec.auth @@ -114,7 +115,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | `forwardAccessToken` | *bool | No | `false` | When `enforceAtGateway: true`, forward the user's OAuth access token to the upstream service via the `Authorization: Bearer ` header. Use when the app needs to read the JWT itself (e.g., to inspect the `groups` claim for per-user authorization). Without this, the gateway only stores the token in an encrypted session cookie that the backend cannot decode. | | `denyRedirect` | [][DenyRedirectHeader](#denyredirectheader) | No | - | Headers that, when matched, prevent the OIDC filter from redirecting to the IdP and instead return 401. Helps avoid PKCE race conditions when SPAs fire multiple parallel requests on page load. Only applies when `enforceAtGateway: true`. | | `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | -| `clientSecretRef` | string | No | - | Name (string) of a Secret in the same namespace containing keys `client-id` and `client-secret`. **Note:** the spec field is a plain string (the Secret name), not the `{name, namespace}` object reference used by `status.clientSecretRef`. If omitted and `provisionClient` is true, the operator creates a Secret named `-oidc-client` with keys: `client-id`, `client-secret`, and `issuer-url`. | +| `clientSecretRef` | string | No | - | **Accepted by the API but ignored by operator v0.1.1.** The operator always reads and writes the Secret named `-oidc-client` (keys `client-id`, `client-secret`, `issuer-url`). If you manage credentials yourself (`provisionClient: false`), create the Secret under that name. | | `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | | `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](auth-flow.md#reading-user-identity-in-your-app)). | | `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | @@ -177,12 +178,14 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `enabled` | bool | No | `false` | Whether this service appears on the Nebari landing page. Set to `true` to opt in. | -| `displayName` | string | No (required when enabled) | - | Human-readable name on the landing page. Max 64 chars. | +| `displayName` | string | No | - | Human-readable name on the landing page. Max 64 chars. Set it whenever `enabled` is `true`; the operator does not reject a missing value. | | `description` | string | No | - | Supplementary text on the service card. Max 256 chars. | -| `icon` | string | No | - | Icon identifier or URL. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | +| `icon` | string | No | - | Icon identifier or URL, shown in both light and dark mode unless `iconLight`/`iconDark` are set. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | +| `iconLight` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in light mode. Takes precedence over `icon`. | +| `iconDark` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in dark mode. Takes precedence over `icon`. | | `category` | string | No | - | Group services together. Common categories: `Development`, `Monitoring`, `Platform`, `Data Science`. | | `priority` | *int | No | `100` | Sort order within a category (lower = higher priority). Range: 0-1000. | -| `externalUrl` | string | No | `https://` | Override the default URL. | +| `externalUrl` | string | No | `https://` (`http://` when `routing.tls.enabled: false`) | Override the default URL. | | `healthCheck` | [HealthCheckConfig](#speclandingpagehealthcheck) | No | - | Health-status monitoring for the service card. | ### spec.landingPage.healthCheck @@ -205,24 +208,44 @@ The operator writes status conditions and several status fields for downstream c |-----------|-------------| | `RoutingReady` | HTTPRoute has been created and the Gateway is routing traffic. | | `TLSReady` | TLS certificate is provisioned and the HTTPS listener is configured. | -| `AuthReady` | SecurityPolicy is created and the OIDC client is available. Only set when `auth.enabled=true`. | -| `Ready` | Aggregate condition - all components are ready. | +| `AuthReady` | SecurityPolicy is created and the OIDC client is available. `False` with reason `AuthDisabled` when auth is off. | +| `Ready` | Set from the core checks (namespace label, Service, validation) and from hard reconcile failures. It does **not** wait for the other three: a NebariApp with `RoutingReady=False/RoutingNotConfigured` or `TLSReady=False/TLSDisabled` still reports `Ready=True`. Check the specific conditions you depend on. | ### Condition Reasons -| Reason | Description | -|--------|-------------| -| `Available` | Resource is functioning correctly. | -| `Reconciling` | Reconciliation is in progress. | -| `ReconcileSuccess` | Reconciliation completed successfully. | -| `ValidationSuccess` | Validation passed successfully. | -| `Failed` | Reconciliation failed. | -| `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | -| `ServiceNotFound` | The referenced Service does not exist. | -| `SecretNotFound` | The referenced Secret does not exist. | -| `GatewayNotFound` | The target Gateway does not exist. | -| `GatewayListenerConflict` | The per-app Gateway listener conflicts with another NebariApp on the same hostname. | -| `CertificateNotReady` | The cert-manager Certificate is not yet ready. | +These are the reasons the v0.1.1 controllers set, grouped by condition. + +| Condition | Reason | Meaning | +|-----------|--------|---------| +| `Ready` | `Reconciling` | Reconciliation is in progress. | +| `Ready` | `ValidationSuccess` | Core validation passed. | +| `Ready` | `ReconcileSuccess` | Reconciliation completed. Does not imply routing, TLS or auth are ready. | +| `Ready` | `Failed` | A reconcile step failed; the message names it. | +| `Ready` | `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | +| `Ready` | `ServiceNotFound` | The referenced Service does not exist. | +| `Ready` | `ResourceNameTooLong` | A generated resource name would exceed Kubernetes limits. Shorten the NebariApp name. | +| `RoutingReady` | `HTTPRouteCreated` / `HTTPRouteReady` | The HTTPRoute exists and is configured. | +| `RoutingReady` | `RoutingNotConfigured` | `spec.routing` is unset, so no HTTPRoute is created. | +| `RoutingReady` | `BuildFailed` / `CreationFailed` / `UpdateFailed` | The HTTPRoute could not be built, created or updated. | +| `TLSReady` | `TLSConfigured` | Certificate and HTTPS listener are in place. | +| `TLSReady` | `TLSDisabled` | TLS is off: `routing.tls.enabled: false`, or `spec.routing` is unset. | +| `TLSReady` | `ClusterIssuerNotConfigured` | The operator has no cert-manager ClusterIssuer configured and no `tls.secretName` was given. | +| `TLSReady` | `CertificateNotReady` | The cert-manager Certificate exists but is not ready yet. | +| `TLSReady` | `CertificateFailed` / `CertificateCheckFailed` / `CertificateCleanupFailed` | Creating, checking or removing the Certificate failed. | +| `TLSReady` | `GatewayListenerFailed` | The per-app HTTPS listener could not be added to the Gateway. | +| `TLSReady` | `GatewayListenerConflict` | Another NebariApp already owns a listener for this hostname. | +| `TLSReady` | `UserProvidedSecretReady` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret exists and has type `kubernetes.io/tls`. | +| `TLSReady` | `UserProvidedSecretNotFound` / `UserProvidedSecretInvalidType` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret is missing or is not `kubernetes.io/tls`. | +| `TLSReady` | `UserProvidedSecretCheckFailed` | *(v0.1.0-alpha.20+)* The Secret could not be checked (transient API or RBAC error). | +| `AuthReady` | `AuthConfigured` | OIDC client and SecurityPolicy are configured. | +| `AuthReady` | `AuthDisabled` | `auth.enabled` is `false`. | +| `AuthReady` | `InvalidProvider` / `ValidationFailed` | The auth configuration is invalid; the message says why. | +| `AuthReady` | `ProvisioningFailed` / `ProvisioningNotSupported` | The OIDC client could not be provisioned, or the provider does not support provisioning. | +| `AuthReady` | `RBACFailed` | The Role/RoleBinding for the OIDC Secret could not be created. | +| `AuthReady` | `SecurityPolicyFailed` / `SecurityPolicyCleanupFailed` | Creating or removing the SecurityPolicy failed. | +| `AuthReady` | `TokenExchangeFailed` | Token exchange policies could not be configured. | + +The API also defines `Available`, `SecretNotFound` and `GatewayNotFound`, but v0.1.1 never sets them. ### Status Fields @@ -230,10 +253,10 @@ The operator writes status conditions and several status fields for downstream c |-------|------|-------------| | `observedGeneration` | int64 | Most recent `metadata.generation` observed by the controller. | | `hostname` | string | Mirror of `spec.hostname` for easy reference. | -| `gatewayRef` | object `{name, namespace}` | The Gateway resource routing traffic to this app. | -| `clientSecretRef` | object `{name, namespace}` | The Secret containing OIDC client credentials. | +| `gatewayRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. | +| `clientSecretRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. The Secret is always `-oidc-client`. | | `authConfigHash` | string | SHA-256 hash of the last successfully provisioned OIDC client config. Used to skip re-provisioning when the spec is unchanged. To force re-provisioning, set the `nebari.dev/force-reprovision` annotation; the operator removes it once the forced re-provision completes. | -| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `category`, `priority`, `visibility`, `requiredGroups`. | +| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `iconLight`, `iconDark`, `category`, `priority`, `visibility`, `requiredGroups`. `requiredGroups` is informational; see `spec.auth.groups`. | ## Namespace Opt-In @@ -362,10 +385,6 @@ nebariapp: enabled: false provider: keycloak provisionClient: true - scopes: - - openid - - profile - - email gateway: public ``` diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 72e1d08..7f26c7f 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -102,6 +102,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `enabled` | *bool | No | `true` | Whether to provision a TLS certificate via cert-manager and configure an HTTPS listener on the Gateway. When `false`, only HTTP listeners are used. | +| `secretName` | string | No | - | *(v0.1.0-alpha.20+)* Name of a pre-existing `kubernetes.io/tls` Secret in `envoy-gateway-system` to use for the HTTPS listener instead of a cert-manager Certificate (for wildcard or externally managed certs). When set, the operator creates no Certificate and cleans up any it owns. You create and rotate the Secret. Ignored when `enabled` is `false`. Max 253 chars, DNS-subdomain format. | ## spec.auth @@ -116,7 +117,7 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | `forwardAccessToken` | *bool | No | `false` | When `enforceAtGateway: true`, forward the user's OAuth access token to the upstream service via the `Authorization: Bearer ` header. Use when the app needs to read the JWT itself (e.g., to inspect the `groups` claim for per-user authorization). Without this, the gateway only stores the token in an encrypted session cookie that the backend cannot decode. | | `denyRedirect` | [][DenyRedirectHeader](#denyredirectheader) | No | - | Headers that, when matched, prevent the OIDC filter from redirecting to the IdP and instead return 401. Helps avoid PKCE race conditions when SPAs fire multiple parallel requests on page load. Only applies when `enforceAtGateway: true`. | | `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | -| `clientSecretRef` | string | No | - | Name (string) of a Secret in the same namespace containing keys `client-id` and `client-secret`. **Note:** the spec field is a plain string (the Secret name), not the `{name, namespace}` object reference used by `status.clientSecretRef`. If omitted and `provisionClient` is true, the operator creates a Secret named `-oidc-client` with keys: `client-id`, `client-secret`, and `issuer-url`. | +| `clientSecretRef` | string | No | - | **Accepted by the API but ignored by operator v0.1.1.** The operator always reads and writes the Secret named `-oidc-client` (keys `client-id`, `client-secret`, `issuer-url`). If you manage credentials yourself (`provisionClient: false`), create the Secret under that name. | | `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | | `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](/auth-flow/#reading-user-identity-in-your-app)). | | `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | @@ -179,12 +180,14 @@ Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `enabled` | bool | No | `false` | Whether this service appears on the Nebari landing page. Set to `true` to opt in. | -| `displayName` | string | No (required when enabled) | - | Human-readable name on the landing page. Max 64 chars. | +| `displayName` | string | No | - | Human-readable name on the landing page. Max 64 chars. Set it whenever `enabled` is `true`; the operator does not reject a missing value. | | `description` | string | No | - | Supplementary text on the service card. Max 256 chars. | -| `icon` | string | No | - | Icon identifier or URL. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | +| `icon` | string | No | - | Icon identifier or URL, shown in both light and dark mode unless `iconLight`/`iconDark` are set. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | +| `iconLight` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in light mode. Takes precedence over `icon`. | +| `iconDark` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in dark mode. Takes precedence over `icon`. | | `category` | string | No | - | Group services together. Common categories: `Development`, `Monitoring`, `Platform`, `Data Science`. | | `priority` | *int | No | `100` | Sort order within a category (lower = higher priority). Range: 0-1000. | -| `externalUrl` | string | No | `https://` | Override the default URL. | +| `externalUrl` | string | No | `https://` (`http://` when `routing.tls.enabled: false`) | Override the default URL. | | `healthCheck` | [HealthCheckConfig](#speclandingpagehealthcheck) | No | - | Health-status monitoring for the service card. | ### spec.landingPage.healthCheck @@ -208,24 +211,44 @@ The operator writes status conditions and several status fields for downstream c |-----------|-------------| | `RoutingReady` | HTTPRoute has been created and the Gateway is routing traffic. | | `TLSReady` | TLS certificate is provisioned and the HTTPS listener is configured. | -| `AuthReady` | SecurityPolicy is created and the OIDC client is available. Only set when `auth.enabled=true`. | -| `Ready` | Aggregate condition - all components are ready. | +| `AuthReady` | SecurityPolicy is created and the OIDC client is available. `False` with reason `AuthDisabled` when auth is off. | +| `Ready` | Set from the core checks (namespace label, Service, validation) and from hard reconcile failures. It does **not** wait for the other three: a NebariApp with `RoutingReady=False/RoutingNotConfigured` or `TLSReady=False/TLSDisabled` still reports `Ready=True`. Check the specific conditions you depend on. | ### Condition Reasons -| Reason | Description | -|--------|-------------| -| `Available` | Resource is functioning correctly. | -| `Reconciling` | Reconciliation is in progress. | -| `ReconcileSuccess` | Reconciliation completed successfully. | -| `ValidationSuccess` | Validation passed successfully. | -| `Failed` | Reconciliation failed. | -| `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | -| `ServiceNotFound` | The referenced Service does not exist. | -| `SecretNotFound` | The referenced Secret does not exist. | -| `GatewayNotFound` | The target Gateway does not exist. | -| `GatewayListenerConflict` | The per-app Gateway listener conflicts with another NebariApp on the same hostname. | -| `CertificateNotReady` | The cert-manager Certificate is not yet ready. | +These are the reasons the v0.1.1 controllers set, grouped by condition. + +| Condition | Reason | Meaning | +|-----------|--------|---------| +| `Ready` | `Reconciling` | Reconciliation is in progress. | +| `Ready` | `ValidationSuccess` | Core validation passed. | +| `Ready` | `ReconcileSuccess` | Reconciliation completed. Does not imply routing, TLS or auth are ready. | +| `Ready` | `Failed` | A reconcile step failed; the message names it. | +| `Ready` | `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | +| `Ready` | `ServiceNotFound` | The referenced Service does not exist. | +| `Ready` | `ResourceNameTooLong` | A generated resource name would exceed Kubernetes limits. Shorten the NebariApp name. | +| `RoutingReady` | `HTTPRouteCreated` / `HTTPRouteReady` | The HTTPRoute exists and is configured. | +| `RoutingReady` | `RoutingNotConfigured` | `spec.routing` is unset, so no HTTPRoute is created. | +| `RoutingReady` | `BuildFailed` / `CreationFailed` / `UpdateFailed` | The HTTPRoute could not be built, created or updated. | +| `TLSReady` | `TLSConfigured` | Certificate and HTTPS listener are in place. | +| `TLSReady` | `TLSDisabled` | TLS is off: `routing.tls.enabled: false`, or `spec.routing` is unset. | +| `TLSReady` | `ClusterIssuerNotConfigured` | The operator has no cert-manager ClusterIssuer configured and no `tls.secretName` was given. | +| `TLSReady` | `CertificateNotReady` | The cert-manager Certificate exists but is not ready yet. | +| `TLSReady` | `CertificateFailed` / `CertificateCheckFailed` / `CertificateCleanupFailed` | Creating, checking or removing the Certificate failed. | +| `TLSReady` | `GatewayListenerFailed` | The per-app HTTPS listener could not be added to the Gateway. | +| `TLSReady` | `GatewayListenerConflict` | Another NebariApp already owns a listener for this hostname. | +| `TLSReady` | `UserProvidedSecretReady` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret exists and has type `kubernetes.io/tls`. | +| `TLSReady` | `UserProvidedSecretNotFound` / `UserProvidedSecretInvalidType` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret is missing or is not `kubernetes.io/tls`. | +| `TLSReady` | `UserProvidedSecretCheckFailed` | *(v0.1.0-alpha.20+)* The Secret could not be checked (transient API or RBAC error). | +| `AuthReady` | `AuthConfigured` | OIDC client and SecurityPolicy are configured. | +| `AuthReady` | `AuthDisabled` | `auth.enabled` is `false`. | +| `AuthReady` | `InvalidProvider` / `ValidationFailed` | The auth configuration is invalid; the message says why. | +| `AuthReady` | `ProvisioningFailed` / `ProvisioningNotSupported` | The OIDC client could not be provisioned, or the provider does not support provisioning. | +| `AuthReady` | `RBACFailed` | The Role/RoleBinding for the OIDC Secret could not be created. | +| `AuthReady` | `SecurityPolicyFailed` / `SecurityPolicyCleanupFailed` | Creating or removing the SecurityPolicy failed. | +| `AuthReady` | `TokenExchangeFailed` | Token exchange policies could not be configured. | + +The API also defines `Available`, `SecretNotFound` and `GatewayNotFound`, but v0.1.1 never sets them. ### Status Fields @@ -233,10 +256,10 @@ The operator writes status conditions and several status fields for downstream c |-------|------|-------------| | `observedGeneration` | int64 | Most recent `metadata.generation` observed by the controller. | | `hostname` | string | Mirror of `spec.hostname` for easy reference. | -| `gatewayRef` | object `{name, namespace}` | The Gateway resource routing traffic to this app. | -| `clientSecretRef` | object `{name, namespace}` | The Secret containing OIDC client credentials. | +| `gatewayRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. | +| `clientSecretRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. The Secret is always `-oidc-client`. | | `authConfigHash` | string | SHA-256 hash of the last successfully provisioned OIDC client config. Used to skip re-provisioning when the spec is unchanged. To force re-provisioning, set the `nebari.dev/force-reprovision` annotation; the operator removes it once the forced re-provision completes. | -| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `category`, `priority`, `visibility`, `requiredGroups`. | +| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `iconLight`, `iconDark`, `category`, `priority`, `visibility`, `requiredGroups`. `requiredGroups` is informational; see `spec.auth.groups`. | ## Namespace Opt-In @@ -365,10 +388,6 @@ nebariapp: enabled: false provider: keycloak provisionClient: true - scopes: - - openid - - profile - - email gateway: public ``` From 592a497ea884f725408729c2cdfbe07c69410359 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 1 Oct 2026 14:50:10 +0200 Subject: [PATCH 10/16] Match the CRD reference Helm section to #44 so the PRs merge cleanly (#41) #44 rewrites the same Helm section in both copies of the CRD reference. Use its text, with the review suggestions posted on #44 applied (fetch step, accurate toJson rule, routing block), so both PRs make identical changes there and merge in either order. Drops two sentences #45 had added that #44 does not carry: the link to the nebari-app chart directory and the note that required fields are checked after rendering. --- docs/nebariapp-crd-reference.md | 4 ++++ docs/site/src/content/docs/nebariapp-crd-reference.md | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 286adf7..c0d288d 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -385,6 +385,10 @@ nebariapp: enabled: false provider: keycloak provisionClient: true + scopes: + - openid + - profile + - email gateway: public ``` diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 7f26c7f..671b004 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -388,6 +388,10 @@ nebariapp: enabled: false provider: keycloak provisionClient: true + scopes: + - openid + - profile + - email gateway: public ``` From 9589ce995d03faa302d7827dd4caa8a62d371333 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Fri, 2 Oct 2026 15:33:34 +0200 Subject: [PATCH 11/16] ci: wait for pods to terminate before reusing the app=my-pack label Deleting the vanilla-yaml Deployment with the default background cascade returned before its pods were gone, so the kustomize step's 'kubectl wait -l app=my-pack' matched the terminating pod and failed with NotFound. Use foreground cascading so cleanup blocks until pods are deleted. --- .github/workflows/test-integration.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index f51bf2d..5e9b677 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -144,7 +144,8 @@ jobs: run: | kubectl delete nebariapp my-pack --ignore-not-found --wait --timeout=60s kubectl delete -f examples/vanilla-yaml/deployment.yaml \ - -f examples/vanilla-yaml/service.yaml --ignore-not-found --wait + -f examples/vanilla-yaml/service.yaml --ignore-not-found --wait \ + --cascade=foreground --timeout=60s # ============================================================ # Test: kustomize-nginx (production overlay - auth enabled) @@ -159,7 +160,7 @@ jobs: - name: "Cleanup kustomize-nginx" if: always() - run: kubectl delete -k examples/kustomize-nginx/overlays/production/ --ignore-not-found --wait --timeout=60s + run: kubectl delete -k examples/kustomize-nginx/overlays/production/ --ignore-not-found --wait --cascade=foreground --timeout=60s # ============================================================ # Test: basic-nginx (Helm, no auth) From a26aa28e92641ad5881c5f573a414f41ca6f9ca2 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Wed, 7 Oct 2026 12:14:19 +0200 Subject: [PATCH 12/16] Retire the hand-maintained NebariApp field tables (#30) Both copies of the CRD reference carried a hand-maintained field reference that duplicates the operator's generated api-reference.md. Replace the field, status and condition tables with a pointer to the generated reference, the reconciler docs and the nebari-app chart guide, all pinned to v0.1.1, plus an index of the examples. Keep what only this repository documents: namespace opt-in, who can read the OIDC Secret, and the plain YAML, Kustomize and Helm deployment patterns. Keep the two caveats the generated reference gets wrong or omits: auth.groups is not enforced in v0.1.1 (nebari-operator#153), and iconLight/iconDark are not in the alpha.20 operator NIC v0.14.0 deploys. Repoint the nine references that promised a complete field reference on this page (README, both auth-flow copies, index, build-your-own, what-is-a-software-pack) to the generated reference. Links to the Pack Specification are left out until nebari-operator#187 publishes it. Supersedes #30. Refs nebari-dev/governance#59 --- README.md | 8 +- docs/auth-flow.md | 6 +- docs/nebariapp-crd-reference.md | 287 +++-------------- docs/site/src/content/docs/auth-flow.md | 2 +- docs/site/src/content/docs/build-your-own.md | 2 +- docs/site/src/content/docs/index.md | 2 +- .../content/docs/nebariapp-crd-reference.md | 288 +++--------------- .../content/docs/what-is-a-software-pack.md | 2 +- 8 files changed, 91 insertions(+), 506 deletions(-) diff --git a/README.md b/README.md index aa87c08..972d974 100644 --- a/README.md +++ b/README.md @@ -183,7 +183,7 @@ software-pack-template/ login-test.sh # Logs in with curl and checks the app .cache/ # (gitignored) Cloned nebari-operator scripts docs/ - nebariapp-crd-reference.md # Full NebariApp field reference + nebariapp-crd-reference.md # Where the field reference lives, plus deployment patterns auth-flow.md # Authentication flow details .gitignore .editorconfig @@ -304,8 +304,8 @@ name: '{{ include "my-pack.fullname" . }}' # fails to render ### Beyond the basics The fields shown above cover the common cases. The operator also supports several -more specialized features. Each is documented in -[docs/nebariapp-crd-reference.md](docs/nebariapp-crd-reference.md): +more specialized features. Each is documented in the +[NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md), generated from the operator's Go types: - **`routing.publicRoutes`** - paths that bypass OIDC auth (e.g., `/healthz`, webhooks, public APIs). @@ -331,7 +331,7 @@ more specialized features. Each is documented in - **`service.namespace`** - point the NebariApp at a Service in a different namespace. -For the complete field reference, see [docs/nebariapp-crd-reference.md](docs/nebariapp-crd-reference.md). +For the complete field reference, see the [NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md). ## Example 1: Vanilla YAML (Plain Manifests) diff --git a/docs/auth-flow.md b/docs/auth-flow.md index 18e1291..b26bdcd 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -398,7 +398,7 @@ for the complete configuration. ## NebariApp CRD vs Envoy Gateway SecurityPolicy -The fields documented in [nebariapp-crd-reference.md](nebariapp-crd-reference.md) are the +The fields documented in the [NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md) are the fields the **operator** understands - they go on `spec.auth` of the NebariApp resource. At runtime, the operator generates an Envoy Gateway `SecurityPolicy` from the NebariApp, and that SecurityPolicy has its own (much larger) set of OIDC tuning knobs. @@ -406,8 +406,8 @@ and that SecurityPolicy has its own (much larger) set of OIDC tuning knobs. This section is specifically about the **Envoy Gateway SecurityPolicy OIDC filter** fields. Other Keycloak-side or client-provisioning features (`spaClient`, `deviceFlowClient`, `keycloakConfig`, `tokenExchange`) are first-class NebariApp -fields and are documented in -[nebariapp-crd-reference.md](nebariapp-crd-reference.md). +fields and are documented in the +[NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md). For the OIDC filter fields specifically, mentally place each one in one of these buckets: diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index c0d288d..0e02eb5 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -1,262 +1,55 @@ # NebariApp CRD Reference -Complete field-by-field reference for the NebariApp custom resource. +Where to find the NebariApp field reference, and how to declare a NebariApp with plain YAML, +Kustomize, or Helm. **API Version:** `reconcilers.nebari.dev/v1` **Kind:** `NebariApp` -**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/api/v1/nebariapp_types.go) -**Operator version this doc tracks:** `v0.1.1`. NIC v0.14.0 deploys `v0.1.0-alpha.20`; fields -added after that release are marked *(v0.1.0+)*. +**Operator version this page tracks:** `v0.1.1` -## Full Example +This page used to hold a hand-maintained copy of the NebariApp field reference. It went stale: +it tracked operator `v0.1.0-alpha.19` while the operator moved on, and nothing detected the +drift. Two copies of it lived in this repository, and the operator generates a third from the +Go types. The field reference now has one home, next to the code that enforces it. -```yaml -apiVersion: reconcilers.nebari.dev/v1 -kind: NebariApp -metadata: - name: my-pack - namespace: my-pack -spec: - hostname: my-pack.nebari.example.com - serviceAccountName: my-pack - service: - name: my-pack - port: 80 - routing: - routes: - - pathPrefix: / - pathType: PathPrefix - publicRoutes: - - pathPrefix: /healthz - pathType: Exact - tls: - enabled: true - annotations: - argocd.argoproj.io/tracking-id: my-pack - auth: - enabled: true - provider: keycloak - provisionClient: true - enforceAtGateway: true - forwardAccessToken: false - # redirectURI defaults to /oauth2/callback - omit to use the operator default - scopes: - - openid - - profile - - email - # groups is NOT enforced by the gateway in v0.1.1 - see spec.auth below - groups: - - admin - gateway: public - landingPage: - enabled: true - displayName: My Pack - description: A short description for the landing page card. - icon: jupyter - category: Development - priority: 100 -``` +## Where to look instead + +| For | Go to | +| --- | ----- | +| Every field, type, and default | [NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md), generated from the Go types | +| How the operator reconciles a NebariApp | [operator reconciler docs](https://github.com/nebari-dev/nebari-operator/tree/v0.1.1/docs/reconcilers) | +| Step-by-step onboarding with the `nebari-app` chart | [Onboarding an app](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/using-the-nebari-app-chart.md) | + +The API reference is generated by `make docs` in the operator repository, so it cannot fall +behind the types the way this page did. -## spec +## What the generated reference does not tell you -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `hostname` | string | Yes | - | FQDN where the app will be accessible. Used to generate the HTTPRoute and TLS certificate. Must match pattern `^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`. | -| `service` | [ServiceReference](#specservice) | Yes | - | The backend Kubernetes Service that receives traffic. | -| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | -| `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | -| `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | -| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. When the operator provisions an OIDC client, it creates a Role and RoleBinding that let this ServiceAccount `get` the OIDC client Secret through the Kubernetes API. This grants access; it does not restrict anyone else. See [who can read the OIDC Secret](#who-can-read-the-oidc-secret). | -| `landingPage` | [LandingPageConfig](#speclandingpage) | No | - | Controls how this service appears on the Nebari landing page. | +- **`auth.groups` is not enforced in v0.1.1.** The generated reference says only members of + the listed groups are authorized. They are not. The operator creates the groups in Keycloak + and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the + SecurityPolicy it generates contains no authorization rule, so any user who can log in to the + realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group + today, verify the token in your app and check its `groups` claim (see + [Authentication Flow](auth-flow.md#reading-user-identity-in-your-app)). +- **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` + and `landingPage.iconDark` were added after alpha.20 and are not available there. -## spec.service +## What this repository gives you -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Name of the Kubernetes Service. | -| `port` | int32 | Yes | - | Port number on the Service to route traffic to. Range: 1-65535. | -| `namespace` | string | No | NebariApp's namespace | Namespace of the Service. Allows referencing services in other namespaces for centralized service architectures. The operator has cluster-scoped read permission on Services. | +Worked examples. Each `examples/` directory onboards the same app a different way: -## spec.routing +| Example | Shows | +| ------- | ----- | +| `examples/vanilla-yaml` | The plainest possible `NebariApp`, no templating | +| `examples/basic-nginx` | A minimal Helm chart that renders one | +| `examples/kustomize-nginx` | A Kustomize base with dev and production overlays | +| `examples/wrap-existing-chart` | Wrapping an upstream chart you do not control | +| `examples/auth-fastapi` | An app verifying the identity token the platform provides | -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `routes` | [][RouteMatch](#routematch) | No | - | Path-based routing rules. If omitted, all traffic to the hostname is routed to the service. | -| `publicRoutes` | [][RouteMatch](#routematch) | No | - | Paths that bypass OIDC authentication. When `auth.enabled=true` and `auth.enforceAtGateway=true`, these paths are routed via a separate HTTPRoute that is not protected by the SecurityPolicy. Default `pathType` is `Exact` here (vs `PathPrefix` for `routes`) - safer for auth bypass. | -| `tls` | [RoutingTLSConfig](#specroutingtls) | No | - | TLS certificate management configuration. | -| `annotations` | map[string]string | No | - | Additional annotations merged onto the generated HTTPRoute. Useful for tools like ArgoCD that track resources via annotations (e.g. `argocd.argoproj.io/tracking-id`). Operator-managed annotations always take precedence. | - -### RouteMatch - -Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `pathPrefix` | string | Yes | - | Path to match. Must start with `/`. Examples: `/`, `/api/v1`, `/dashboard`. | -| `pathType` | string | No | `PathPrefix` (in `routes`), `Exact` (in `publicRoutes`) | How the path is matched. Values: `PathPrefix`, `Exact`. | - -### spec.routing.tls - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | *bool | No | `true` | Whether to provision a TLS certificate via cert-manager and configure an HTTPS listener on the Gateway. When `false`, only HTTP listeners are used. | -| `secretName` | string | No | - | *(v0.1.0-alpha.20+)* Name of a pre-existing `kubernetes.io/tls` Secret in `envoy-gateway-system` to use for the HTTPS listener instead of a cert-manager Certificate (for wildcard or externally managed certs). When set, the operator creates no Certificate and cleans up any it owns. You create and rotate the Secret. Ignored when `enabled` is `false`. Max 253 chars, DNS-subdomain format. | - -## spec.auth - -> **Validation rule:** `forwardAccessToken: true` requires `enforceAtGateway: true`. - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether to enforce OIDC authentication. | -| `provider` | string | No | `"keycloak"` | OIDC provider. Values: `keycloak`, `generic-oidc`. | -| `provisionClient` | *bool | No | `true` | Auto-provision an OIDC client in the provider. Only supported for `keycloak`. The operator creates the client and stores credentials in a Secret named `-oidc-client`. The client ID follows the convention `-`. See [auth-flow.md](auth-flow.md#2-kubernetes-secret) for the full secret structure. | -| `enforceAtGateway` | *bool | No | `true` | Create an Envoy Gateway SecurityPolicy for gateway-level auth. When `false`, the operator provisions the client and Secret but does NOT create a SecurityPolicy - the app handles OAuth natively. See [auth-flow.md](auth-flow.md#app-native-oauth) for wiring guidance. | -| `forwardAccessToken` | *bool | No | `false` | When `enforceAtGateway: true`, forward the user's OAuth access token to the upstream service via the `Authorization: Bearer ` header. Use when the app needs to read the JWT itself (e.g., to inspect the `groups` claim for per-user authorization). Without this, the gateway only stores the token in an encrypted session cookie that the backend cannot decode. | -| `denyRedirect` | [][DenyRedirectHeader](#denyredirectheader) | No | - | Headers that, when matched, prevent the OIDC filter from redirecting to the IdP and instead return 401. Helps avoid PKCE race conditions when SPAs fire multiple parallel requests on page load. Only applies when `enforceAtGateway: true`. | -| `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | -| `clientSecretRef` | string | No | - | **Accepted by the API but ignored by operator v0.1.1.** The operator always reads and writes the Secret named `-oidc-client` (keys `client-id`, `client-secret`, `issuer-url`). If you manage credentials yourself (`provisionClient: false`), create the Secret under that name. | -| `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | -| `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](auth-flow.md#reading-user-identity-in-your-app)). | -| `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | -| `spaClient` | [SPAClientConfig](#specauthspaclient) | No | - | Provisions a public Keycloak client for browser-based PKCE flows (e.g., React apps using `keycloak-js`). Distinct from the confidential client used by gateway-enforced auth. | -| `deviceFlowClient` | [DeviceFlowClientConfig](#specauthdeviceflowclient) | No | - | Provisions a public Keycloak client for the OAuth2 Device Authorization Grant (RFC 8628), for CLI/native apps. | -| `keycloakConfig` | [KeycloakClientConfig](#specauthkeycloakconfig) | No | - | Keycloak-specific configuration: realm groups (with optional member assignments) and client-level protocol mappers. Only applied when `provider=keycloak` and `provisionClient=true`. | -| `tokenExchange` | [TokenExchangeConfig](#specauthtokenexchange) | No | - | OAuth 2.0 Token Exchange (RFC 8693). When enabled, other NebariApp clients in the same Keycloak realm can exchange their access tokens for tokens with this client's audience. Requires `KC_FEATURES=token-exchange` on the Keycloak server. | - -### DenyRedirectHeader - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Header name to match against. | -| `value` | string | Yes | - | Header value to match. | -| `type` | string | No | `Exact` | Match type. Values: `Exact`, `Prefix`, `Suffix`, `RegularExpression`. | - -### spec.auth.spaClient - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Provision a public OIDC client for SPA use (PKCE, no client secret). The public client ID is written to the OIDC Secret as `spa-client-id`. | -| `clientId` | string | No | `--spa` | Override the generated client ID. | - -### spec.auth.deviceFlowClient - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Provision a public OIDC client configured for the Device Authorization Grant. The client ID is written to the OIDC Secret as `device-client-id`. | - -### spec.auth.keycloakConfig - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `groups` | [][KeycloakGroup](#keycloakgroup) | No | - | Groups to ensure exist in the realm, with optional user membership assignments. | -| `protocolMappers` | [][KeycloakProtocolMapperConfig](#keycloakprotocolmapperconfig) | No | - | Client-level protocol mappers applied directly to the OIDC client. When specified, the operator's default mappers (e.g., group-membership) are not auto-created. | - -#### KeycloakGroup - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Group name to create in Keycloak. | -| `members` | []string | No | - | Keycloak usernames to add to the group. Membership sync is **additive-only** - users not in this list are not removed. | - -#### KeycloakProtocolMapperConfig - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Protocol mapper name. | -| `protocolMapper` | string | Yes | - | Mapper type identifier (e.g., `oidc-group-membership-mapper`). | -| `config` | map[string]string | No | - | Mapper configuration as arbitrary key-value pairs. Keys/values are mapper-type specific. | - -### spec.auth.tokenExchange - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Enable authorization services on the Keycloak client and create policies allowing other NebariApp clients in the same realm to exchange tokens for this client's audience. | - -## spec.landingPage - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether this service appears on the Nebari landing page. Set to `true` to opt in. | -| `displayName` | string | No | - | Human-readable name on the landing page. Max 64 chars. Set it whenever `enabled` is `true`; the operator does not reject a missing value. | -| `description` | string | No | - | Supplementary text on the service card. Max 256 chars. | -| `icon` | string | No | - | Icon identifier or URL, shown in both light and dark mode unless `iconLight`/`iconDark` are set. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | -| `iconLight` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in light mode. Takes precedence over `icon`. | -| `iconDark` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in dark mode. Takes precedence over `icon`. | -| `category` | string | No | - | Group services together. Common categories: `Development`, `Monitoring`, `Platform`, `Data Science`. | -| `priority` | *int | No | `100` | Sort order within a category (lower = higher priority). Range: 0-1000. | -| `externalUrl` | string | No | `https://` (`http://` when `routing.tls.enabled: false`) | Override the default URL. | -| `healthCheck` | [HealthCheckConfig](#speclandingpagehealthcheck) | No | - | Health-status monitoring for the service card. | - -### spec.landingPage.healthCheck - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether health checks are performed. | -| `path` | string | No | `/health` | HTTP path to check. | -| `port` | *int32 | No | `spec.service.port` | Port to use for health checks. Range: 1-65535. | -| `intervalSeconds` | *int | No | `30` | Health check interval. Range: 10-300. | -| `timeoutSeconds` | *int | No | `5` | Per-check request timeout. Range: 1-30. | - -## Status - -The operator writes status conditions and several status fields for downstream consumers (the webapi watcher, ArgoCD, etc.). - -### Conditions - -| Condition | Description | -|-----------|-------------| -| `RoutingReady` | HTTPRoute has been created and the Gateway is routing traffic. | -| `TLSReady` | TLS certificate is provisioned and the HTTPS listener is configured. | -| `AuthReady` | SecurityPolicy is created and the OIDC client is available. `False` with reason `AuthDisabled` when auth is off. | -| `Ready` | Set from the core checks (namespace label, Service, validation) and from hard reconcile failures. It does **not** wait for the other three: a NebariApp with `RoutingReady=False/RoutingNotConfigured` or `TLSReady=False/TLSDisabled` still reports `Ready=True`. Check the specific conditions you depend on. | - -### Condition Reasons - -These are the reasons the v0.1.1 controllers set, grouped by condition. - -| Condition | Reason | Meaning | -|-----------|--------|---------| -| `Ready` | `Reconciling` | Reconciliation is in progress. | -| `Ready` | `ValidationSuccess` | Core validation passed. | -| `Ready` | `ReconcileSuccess` | Reconciliation completed. Does not imply routing, TLS or auth are ready. | -| `Ready` | `Failed` | A reconcile step failed; the message names it. | -| `Ready` | `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | -| `Ready` | `ServiceNotFound` | The referenced Service does not exist. | -| `Ready` | `ResourceNameTooLong` | A generated resource name would exceed Kubernetes limits. Shorten the NebariApp name. | -| `RoutingReady` | `HTTPRouteCreated` / `HTTPRouteReady` | The HTTPRoute exists and is configured. | -| `RoutingReady` | `RoutingNotConfigured` | `spec.routing` is unset, so no HTTPRoute is created. | -| `RoutingReady` | `BuildFailed` / `CreationFailed` / `UpdateFailed` | The HTTPRoute could not be built, created or updated. | -| `TLSReady` | `TLSConfigured` | Certificate and HTTPS listener are in place. | -| `TLSReady` | `TLSDisabled` | TLS is off: `routing.tls.enabled: false`, or `spec.routing` is unset. | -| `TLSReady` | `ClusterIssuerNotConfigured` | The operator has no cert-manager ClusterIssuer configured and no `tls.secretName` was given. | -| `TLSReady` | `CertificateNotReady` | The cert-manager Certificate exists but is not ready yet. | -| `TLSReady` | `CertificateFailed` / `CertificateCheckFailed` / `CertificateCleanupFailed` | Creating, checking or removing the Certificate failed. | -| `TLSReady` | `GatewayListenerFailed` | The per-app HTTPS listener could not be added to the Gateway. | -| `TLSReady` | `GatewayListenerConflict` | Another NebariApp already owns a listener for this hostname. | -| `TLSReady` | `UserProvidedSecretReady` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret exists and has type `kubernetes.io/tls`. | -| `TLSReady` | `UserProvidedSecretNotFound` / `UserProvidedSecretInvalidType` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret is missing or is not `kubernetes.io/tls`. | -| `TLSReady` | `UserProvidedSecretCheckFailed` | *(v0.1.0-alpha.20+)* The Secret could not be checked (transient API or RBAC error). | -| `AuthReady` | `AuthConfigured` | OIDC client and SecurityPolicy are configured. | -| `AuthReady` | `AuthDisabled` | `auth.enabled` is `false`. | -| `AuthReady` | `InvalidProvider` / `ValidationFailed` | The auth configuration is invalid; the message says why. | -| `AuthReady` | `ProvisioningFailed` / `ProvisioningNotSupported` | The OIDC client could not be provisioned, or the provider does not support provisioning. | -| `AuthReady` | `RBACFailed` | The Role/RoleBinding for the OIDC Secret could not be created. | -| `AuthReady` | `SecurityPolicyFailed` / `SecurityPolicyCleanupFailed` | Creating or removing the SecurityPolicy failed. | -| `AuthReady` | `TokenExchangeFailed` | Token exchange policies could not be configured. | - -The API also defines `Available`, `SecretNotFound` and `GatewayNotFound`, but v0.1.1 never sets them. - -### Status Fields - -| Field | Type | Description | -|-------|------|-------------| -| `observedGeneration` | int64 | Most recent `metadata.generation` observed by the controller. | -| `hostname` | string | Mirror of `spec.hostname` for easy reference. | -| `gatewayRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. | -| `clientSecretRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. The Secret is always `-oidc-client`. | -| `authConfigHash` | string | SHA-256 hash of the last successfully provisioned OIDC client config. Used to skip re-provisioning when the spec is unchanged. To force re-provisioning, set the `nebari.dev/force-reprovision` annotation; the operator removes it once the forced re-provision completes. | -| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `iconLight`, `iconDark`, `category`, `priority`, `visibility`, `requiredGroups`. `requiredGroups` is informational; see `spec.auth.groups`. | +For the authentication sequence end to end, see [Authentication flow](auth-flow.md). +Requirements for promoting a pack are in the +[release readiness checklist](release-readiness-checklist.md). ## Namespace Opt-In @@ -364,7 +157,7 @@ dependencies: Set any NebariApp `spec` field under `nebariapp:` in `values.yaml`. Everything under `nebariapp:` (except `enabled`) is passed through to the NebariApp spec, -so all fields documented above can be set here. Each `{{ ... }}` value is rendered +so every field in the NebariApp API reference can be set here. Each `{{ ... }}` value is rendered with the chart context and must produce valid JSON, so a template that renders a string ends with `| toJson`. Numbers such as the port below don't need it: diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index 0c0662c..04705f6 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -392,7 +392,7 @@ Your app then configures its OAuth provider using these environment variables. ## NebariApp CRD vs Envoy Gateway SecurityPolicy -The fields documented in the [NebariApp CRD Reference](/nebariapp-crd-reference/) are +The fields documented in the [NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md) are the fields the **operator** understands - they go on `spec.auth` of the NebariApp resource. At runtime, the operator generates an Envoy Gateway `SecurityPolicy` from the NebariApp, and that SecurityPolicy has its own (much larger) set of OIDC tuning diff --git a/docs/site/src/content/docs/build-your-own.md b/docs/site/src/content/docs/build-your-own.md index 7eea9fc..dcc5e68 100644 --- a/docs/site/src/content/docs/build-your-own.md +++ b/docs/site/src/content/docs/build-your-own.md @@ -114,6 +114,6 @@ For a private OCI chart registry, use `type: helm`, `enableOCI: "true"`, and the ## Going deeper -- [NebariApp CRD reference](/nebariapp-crd-reference/) - every field explained +- [NebariApp CRD reference](/nebariapp-crd-reference/) - where every field is documented, and how to declare a NebariApp with YAML, Kustomize, or Helm - [Authentication flow](/auth-flow/) - how OIDC works end-to-end, including verifying the IdToken in your app - [Release readiness](/release-readiness/) - maturity levels and the promotion checklist for official packs diff --git a/docs/site/src/content/docs/index.md b/docs/site/src/content/docs/index.md index 618ea95..8a1f18b 100644 --- a/docs/site/src/content/docs/index.md +++ b/docs/site/src/content/docs/index.md @@ -13,7 +13,7 @@ A software pack is a Kubernetes application bundled with a `NebariApp` custom re ## Reference pages -- **[NebariApp CRD reference](/nebariapp-crd-reference/)** - complete field-by-field reference for the `NebariApp` custom resource +- **[NebariApp CRD reference](/nebariapp-crd-reference/)** - where the generated `NebariApp` field reference lives, and how to declare one with YAML, Kustomize, or Helm - **[Authentication flow](/auth-flow/)** - how OIDC works end-to-end; verifying the IdToken in your app - **[Release readiness](/release-readiness/)** - maturity levels and the promotion checklist for first-party packs diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 671b004..3e3e96c 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -2,264 +2,56 @@ title: NebariApp CRD Reference --- -Complete field-by-field reference for the NebariApp custom resource. +Where to find the NebariApp field reference, and how to declare a NebariApp with plain YAML, +Kustomize, or Helm. **API Version:** `reconcilers.nebari.dev/v1` **Kind:** `NebariApp` -**Source:** [nebari-operator/api/v1/nebariapp_types.go](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/api/v1/nebariapp_types.go) -**Operator version this doc tracks:** `v0.1.1`. NIC v0.14.0 deploys `v0.1.0-alpha.20`; fields -added after that release are marked *(v0.1.0+)*. +**Operator version this page tracks:** `v0.1.1` -## Full Example +This page used to hold a hand-maintained copy of the NebariApp field reference. It went stale: +it tracked operator `v0.1.0-alpha.19` while the operator moved on, and nothing detected the +drift. Two copies of it lived in this repository, and the operator generates a third from the +Go types. The field reference now has one home, next to the code that enforces it. -```yaml -apiVersion: reconcilers.nebari.dev/v1 -kind: NebariApp -metadata: - name: my-pack - namespace: my-pack -spec: - hostname: my-pack.nebari.example.com - serviceAccountName: my-pack - service: - name: my-pack - port: 80 - routing: - routes: - - pathPrefix: / - pathType: PathPrefix - publicRoutes: - - pathPrefix: /healthz - pathType: Exact - tls: - enabled: true - annotations: - argocd.argoproj.io/tracking-id: my-pack - auth: - enabled: true - provider: keycloak - provisionClient: true - enforceAtGateway: true - forwardAccessToken: false - # redirectURI defaults to /oauth2/callback - omit to use the operator default - scopes: - - openid - - profile - - email - # groups is NOT enforced by the gateway in v0.1.1 - see spec.auth below - groups: - - admin - gateway: public - landingPage: - enabled: true - displayName: My Pack - description: A short description for the landing page card. - icon: jupyter - category: Development - priority: 100 -``` +## Where to look instead + +| For | Go to | +| --- | ----- | +| Every field, type, and default | [NebariApp API reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md), generated from the Go types | +| How the operator reconciles a NebariApp | [operator reconciler docs](https://github.com/nebari-dev/nebari-operator/tree/v0.1.1/docs/reconcilers) | +| Step-by-step onboarding with the `nebari-app` chart | [Onboarding an app](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/using-the-nebari-app-chart.md) | + +The API reference is generated by `make docs` in the operator repository, so it cannot fall +behind the types the way this page did. -## spec +## What the generated reference does not tell you -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `hostname` | string | Yes | - | FQDN where the app will be accessible. Used to generate the HTTPRoute and TLS certificate. Must match pattern `^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`. | -| `service` | [ServiceReference](#specservice) | Yes | - | The backend Kubernetes Service that receives traffic. | -| `routing` | [RoutingConfig](#specrouting) | No | - | Routing behavior including path rules, TLS, and HTTPRoute annotations. **Omitting `routing` disables operator-managed routing entirely** - the operator skips HTTPRoute creation and cleans up any existing HTTPRoute. TLS is also considered disabled in that case. The NebariApp still reports `Ready=True` (with `RoutingReady=False/RoutingNotConfigured`), so a pack without `routing` deploys cleanly and is unreachable through the gateway. Include at least `routes: [{pathPrefix: /}]`. | -| `auth` | [AuthConfig](#specauth) | No | - | Authentication/authorization configuration. | -| `gateway` | string | No | `"public"` | Which shared Gateway to use. Valid values: `public`, `internal`. | -| `serviceAccountName` | string | No | NebariApp name | Name of the ServiceAccount used by the app's pods. When the operator provisions an OIDC client, it creates a Role and RoleBinding that let this ServiceAccount `get` the OIDC client Secret through the Kubernetes API. This grants access; it does not restrict anyone else. See [who can read the OIDC Secret](#who-can-read-the-oidc-secret). | -| `landingPage` | [LandingPageConfig](#speclandingpage) | No | - | Controls how this service appears on the Nebari landing page. | +- **`auth.groups` is not enforced in v0.1.1.** The generated reference says only members of + the listed groups are authorized. They are not. The operator creates the groups in Keycloak + and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the + SecurityPolicy it generates contains no authorization rule, so any user who can log in to the + realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group + today, verify the token in your app and check its `groups` claim (see + [Authentication Flow](/auth-flow/#reading-user-identity-in-your-app)). +- **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` + and `landingPage.iconDark` were added after alpha.20 and are not available there. -## spec.service +## What this repository gives you -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Name of the Kubernetes Service. | -| `port` | int32 | Yes | - | Port number on the Service to route traffic to. Range: 1-65535. | -| `namespace` | string | No | NebariApp's namespace | Namespace of the Service. Allows referencing services in other namespaces for centralized service architectures. The operator has cluster-scoped read permission on Services. | +Worked examples. Each `examples/` directory onboards the same app a different way: -## spec.routing +| Example | Shows | +| ------- | ----- | +| `examples/vanilla-yaml` | The plainest possible `NebariApp`, no templating | +| `examples/basic-nginx` | A minimal Helm chart that renders one | +| `examples/kustomize-nginx` | A Kustomize base with dev and production overlays | +| `examples/wrap-existing-chart` | Wrapping an upstream chart you do not control | +| `examples/auth-fastapi` | An app verifying the identity token the platform provides | -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `routes` | [][RouteMatch](#routematch) | No | - | Path-based routing rules. If omitted, all traffic to the hostname is routed to the service. | -| `publicRoutes` | [][RouteMatch](#routematch) | No | - | Paths that bypass OIDC authentication. When `auth.enabled=true` and `auth.enforceAtGateway=true`, these paths are routed via a separate HTTPRoute that is not protected by the SecurityPolicy. Default `pathType` is `Exact` here (vs `PathPrefix` for `routes`) - safer for auth bypass. | -| `tls` | [RoutingTLSConfig](#specroutingtls) | No | - | TLS certificate management configuration. | -| `annotations` | map[string]string | No | - | Additional annotations merged onto the generated HTTPRoute. Useful for tools like ArgoCD that track resources via annotations (e.g. `argocd.argoproj.io/tracking-id`). Operator-managed annotations always take precedence. | - -### RouteMatch - -Used in both `spec.routing.routes[]` and `spec.routing.publicRoutes[]`. - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `pathPrefix` | string | Yes | - | Path to match. Must start with `/`. Examples: `/`, `/api/v1`, `/dashboard`. | -| `pathType` | string | No | `PathPrefix` (in `routes`), `Exact` (in `publicRoutes`) | How the path is matched. Values: `PathPrefix`, `Exact`. | - -### spec.routing.tls - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | *bool | No | `true` | Whether to provision a TLS certificate via cert-manager and configure an HTTPS listener on the Gateway. When `false`, only HTTP listeners are used. | -| `secretName` | string | No | - | *(v0.1.0-alpha.20+)* Name of a pre-existing `kubernetes.io/tls` Secret in `envoy-gateway-system` to use for the HTTPS listener instead of a cert-manager Certificate (for wildcard or externally managed certs). When set, the operator creates no Certificate and cleans up any it owns. You create and rotate the Secret. Ignored when `enabled` is `false`. Max 253 chars, DNS-subdomain format. | - -## spec.auth - -> **Validation rule:** `forwardAccessToken: true` requires `enforceAtGateway: true`. - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether to enforce OIDC authentication. | -| `provider` | string | No | `"keycloak"` | OIDC provider. Values: `keycloak`, `generic-oidc`. | -| `provisionClient` | *bool | No | `true` | Auto-provision an OIDC client in the provider. Only supported for `keycloak`. The operator creates the client and stores credentials in a Secret named `-oidc-client`. The client ID follows the convention `-`. See [Authentication Flow](/auth-flow/#2-kubernetes-secret) for the full secret structure. | -| `enforceAtGateway` | *bool | No | `true` | Create an Envoy Gateway SecurityPolicy for gateway-level auth. When `false`, the operator provisions the client and Secret but does NOT create a SecurityPolicy - the app handles OAuth natively. See [Authentication Flow](/auth-flow/#app-native-oauth) for wiring guidance. | -| `forwardAccessToken` | *bool | No | `false` | When `enforceAtGateway: true`, forward the user's OAuth access token to the upstream service via the `Authorization: Bearer ` header. Use when the app needs to read the JWT itself (e.g., to inspect the `groups` claim for per-user authorization). Without this, the gateway only stores the token in an encrypted session cookie that the backend cannot decode. | -| `denyRedirect` | [][DenyRedirectHeader](#denyredirectheader) | No | - | Headers that, when matched, prevent the OIDC filter from redirecting to the IdP and instead return 401. Helps avoid PKCE race conditions when SPAs fire multiple parallel requests on page load. Only applies when `enforceAtGateway: true`. | -| `redirectURI` | string | No | `"/oauth2/callback"` | OAuth2 callback path. The full URL is `https://`. | -| `clientSecretRef` | string | No | - | **Accepted by the API but ignored by operator v0.1.1.** The operator always reads and writes the Secret named `-oidc-client` (keys `client-id`, `client-secret`, `issuer-url`). If you manage credentials yourself (`provisionClient: false`), create the Secret under that name. | -| `scopes` | []string | No | `["openid", "profile", "email"]` | OIDC scopes to request during authentication. | -| `groups` | []string | No | - | **Not enforced in v0.1.1.** The operator creates these groups in Keycloak and publishes them as `status.serviceDiscovery.requiredGroups` for the landing page, but the SecurityPolicy it generates contains no authorization rule, so any user who can log in to the realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](/auth-flow/#reading-user-identity-in-your-app)). | -| `issuerURL` | string | No | - | OIDC issuer URL. Required when `provider=generic-oidc`, ignored for `keycloak`. Example: `https://accounts.google.com`. | -| `spaClient` | [SPAClientConfig](#specauthspaclient) | No | - | Provisions a public Keycloak client for browser-based PKCE flows (e.g., React apps using `keycloak-js`). Distinct from the confidential client used by gateway-enforced auth. | -| `deviceFlowClient` | [DeviceFlowClientConfig](#specauthdeviceflowclient) | No | - | Provisions a public Keycloak client for the OAuth2 Device Authorization Grant (RFC 8628), for CLI/native apps. | -| `keycloakConfig` | [KeycloakClientConfig](#specauthkeycloakconfig) | No | - | Keycloak-specific configuration: realm groups (with optional member assignments) and client-level protocol mappers. Only applied when `provider=keycloak` and `provisionClient=true`. | -| `tokenExchange` | [TokenExchangeConfig](#specauthtokenexchange) | No | - | OAuth 2.0 Token Exchange (RFC 8693). When enabled, other NebariApp clients in the same Keycloak realm can exchange their access tokens for tokens with this client's audience. Requires `KC_FEATURES=token-exchange` on the Keycloak server. | - -### DenyRedirectHeader - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Header name to match against. | -| `value` | string | Yes | - | Header value to match. | -| `type` | string | No | `Exact` | Match type. Values: `Exact`, `Prefix`, `Suffix`, `RegularExpression`. | - -### spec.auth.spaClient - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Provision a public OIDC client for SPA use (PKCE, no client secret). The public client ID is written to the OIDC Secret as `spa-client-id`. | -| `clientId` | string | No | `--spa` | Override the generated client ID. | - -### spec.auth.deviceFlowClient - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Provision a public Keycloak client configured for the Device Authorization Grant. The client ID is written to the OIDC Secret as `device-client-id`. | - -### spec.auth.keycloakConfig - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `groups` | [][KeycloakGroup](#keycloakgroup) | No | - | Groups to ensure exist in the realm, with optional user membership assignments. | -| `protocolMappers` | [][KeycloakProtocolMapperConfig](#keycloakprotocolmapperconfig) | No | - | Client-level protocol mappers applied directly to the OIDC client. When specified, the operator's default mappers (e.g., group-membership) are not auto-created. | - -#### KeycloakGroup - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Group name to create in Keycloak. | -| `members` | []string | No | - | Keycloak usernames to add to the group. Membership sync is **additive-only** - users not in this list are not removed. | - -#### KeycloakProtocolMapperConfig - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `name` | string | Yes | - | Protocol mapper name. | -| `protocolMapper` | string | Yes | - | Mapper type identifier (e.g., `oidc-group-membership-mapper`). | -| `config` | map[string]string | No | - | Mapper configuration as arbitrary key-value pairs. Keys/values are mapper-type specific. | - -### spec.auth.tokenExchange - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Enable authorization services on the Keycloak client and create policies allowing other NebariApp clients in the same realm to exchange tokens for this client's audience. | - -## spec.landingPage - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether this service appears on the Nebari landing page. Set to `true` to opt in. | -| `displayName` | string | No | - | Human-readable name on the landing page. Max 64 chars. Set it whenever `enabled` is `true`; the operator does not reject a missing value. | -| `description` | string | No | - | Supplementary text on the service card. Max 256 chars. | -| `icon` | string | No | - | Icon identifier or URL, shown in both light and dark mode unless `iconLight`/`iconDark` are set. Built-in icons: `jupyter`, `grafana`, `prometheus`, `keycloak`, `argocd`, `kubernetes`. | -| `iconLight` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in light mode. Takes precedence over `icon`. | -| `iconDark` | string | No | - | *(v0.1.0+)* URL of the icon shown when the UI is in dark mode. Takes precedence over `icon`. | -| `category` | string | No | - | Group services together. Common categories: `Development`, `Monitoring`, `Platform`, `Data Science`. | -| `priority` | *int | No | `100` | Sort order within a category (lower = higher priority). Range: 0-1000. | -| `externalUrl` | string | No | `https://` (`http://` when `routing.tls.enabled: false`) | Override the default URL. | -| `healthCheck` | [HealthCheckConfig](#speclandingpagehealthcheck) | No | - | Health-status monitoring for the service card. | - -### spec.landingPage.healthCheck - -| Field | Type | Required | Default | Description | -|-------|------|----------|---------|-------------| -| `enabled` | bool | No | `false` | Whether health checks are performed. | -| `path` | string | No | `/health` | HTTP path to check. | -| `port` | *int32 | No | `spec.service.port` | Port to use for health checks. Range: 1-65535. | -| `intervalSeconds` | *int | No | `30` | Health check interval. Range: 10-300. | -| `timeoutSeconds` | *int | No | `5` | Per-check request timeout. Range: 1-30. | - -## Status - -The operator writes status conditions and several status fields for downstream consumers -(the webapi watcher, ArgoCD, etc.). - -### Conditions - -| Condition | Description | -|-----------|-------------| -| `RoutingReady` | HTTPRoute has been created and the Gateway is routing traffic. | -| `TLSReady` | TLS certificate is provisioned and the HTTPS listener is configured. | -| `AuthReady` | SecurityPolicy is created and the OIDC client is available. `False` with reason `AuthDisabled` when auth is off. | -| `Ready` | Set from the core checks (namespace label, Service, validation) and from hard reconcile failures. It does **not** wait for the other three: a NebariApp with `RoutingReady=False/RoutingNotConfigured` or `TLSReady=False/TLSDisabled` still reports `Ready=True`. Check the specific conditions you depend on. | - -### Condition Reasons - -These are the reasons the v0.1.1 controllers set, grouped by condition. - -| Condition | Reason | Meaning | -|-----------|--------|---------| -| `Ready` | `Reconciling` | Reconciliation is in progress. | -| `Ready` | `ValidationSuccess` | Core validation passed. | -| `Ready` | `ReconcileSuccess` | Reconciliation completed. Does not imply routing, TLS or auth are ready. | -| `Ready` | `Failed` | A reconcile step failed; the message names it. | -| `Ready` | `NamespaceNotOptedIn` | Namespace is missing the `nebari.dev/managed=true` label. | -| `Ready` | `ServiceNotFound` | The referenced Service does not exist. | -| `Ready` | `ResourceNameTooLong` | A generated resource name would exceed Kubernetes limits. Shorten the NebariApp name. | -| `RoutingReady` | `HTTPRouteCreated` / `HTTPRouteReady` | The HTTPRoute exists and is configured. | -| `RoutingReady` | `RoutingNotConfigured` | `spec.routing` is unset, so no HTTPRoute is created. | -| `RoutingReady` | `BuildFailed` / `CreationFailed` / `UpdateFailed` | The HTTPRoute could not be built, created or updated. | -| `TLSReady` | `TLSConfigured` | Certificate and HTTPS listener are in place. | -| `TLSReady` | `TLSDisabled` | TLS is off: `routing.tls.enabled: false`, or `spec.routing` is unset. | -| `TLSReady` | `ClusterIssuerNotConfigured` | The operator has no cert-manager ClusterIssuer configured and no `tls.secretName` was given. | -| `TLSReady` | `CertificateNotReady` | The cert-manager Certificate exists but is not ready yet. | -| `TLSReady` | `CertificateFailed` / `CertificateCheckFailed` / `CertificateCleanupFailed` | Creating, checking or removing the Certificate failed. | -| `TLSReady` | `GatewayListenerFailed` | The per-app HTTPS listener could not be added to the Gateway. | -| `TLSReady` | `GatewayListenerConflict` | Another NebariApp already owns a listener for this hostname. | -| `TLSReady` | `UserProvidedSecretReady` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret exists and has type `kubernetes.io/tls`. | -| `TLSReady` | `UserProvidedSecretNotFound` / `UserProvidedSecretInvalidType` | *(v0.1.0-alpha.20+)* The `tls.secretName` Secret is missing or is not `kubernetes.io/tls`. | -| `TLSReady` | `UserProvidedSecretCheckFailed` | *(v0.1.0-alpha.20+)* The Secret could not be checked (transient API or RBAC error). | -| `AuthReady` | `AuthConfigured` | OIDC client and SecurityPolicy are configured. | -| `AuthReady` | `AuthDisabled` | `auth.enabled` is `false`. | -| `AuthReady` | `InvalidProvider` / `ValidationFailed` | The auth configuration is invalid; the message says why. | -| `AuthReady` | `ProvisioningFailed` / `ProvisioningNotSupported` | The OIDC client could not be provisioned, or the provider does not support provisioning. | -| `AuthReady` | `RBACFailed` | The Role/RoleBinding for the OIDC Secret could not be created. | -| `AuthReady` | `SecurityPolicyFailed` / `SecurityPolicyCleanupFailed` | Creating or removing the SecurityPolicy failed. | -| `AuthReady` | `TokenExchangeFailed` | Token exchange policies could not be configured. | - -The API also defines `Available`, `SecretNotFound` and `GatewayNotFound`, but v0.1.1 never sets them. - -### Status Fields - -| Field | Type | Description | -|-------|------|-------------| -| `observedGeneration` | int64 | Most recent `metadata.generation` observed by the controller. | -| `hostname` | string | Mirror of `spec.hostname` for easy reference. | -| `gatewayRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. | -| `clientSecretRef` | object `{name, namespace}` | Defined in the API but not written by v0.1.1. The Secret is always `-oidc-client`. | -| `authConfigHash` | string | SHA-256 hash of the last successfully provisioned OIDC client config. Used to skip re-provisioning when the spec is unchanged. To force re-provisioning, set the `nebari.dev/force-reprovision` annotation; the operator removes it once the forced re-provision completes. | -| `serviceDiscovery` | object | URL-resolved view of `spec.landingPage` for the webapi/landing-page watcher. Includes `enabled`, `displayName`, `description`, `url`, `icon`, `iconLight`, `iconDark`, `category`, `priority`, `visibility`, `requiredGroups`. `requiredGroups` is informational; see `spec.auth.groups`. | +For the authentication sequence end to end, see [Authentication flow](/auth-flow/). +Requirements for promoting a pack are in the +[release readiness checklist](/release-readiness/). ## Namespace Opt-In @@ -367,7 +159,7 @@ dependencies: Set any NebariApp `spec` field under `nebariapp:` in `values.yaml`. Everything under `nebariapp:` (except `enabled`) is passed through to the NebariApp spec, -so all fields documented above can be set here. Each `{{ ... }}` value is rendered +so every field in the NebariApp API reference can be set here. Each `{{ ... }}` value is rendered with the chart context and must produce valid JSON, so a template that renders a string ends with `| toJson`. Numbers such as the port below don't need it: diff --git a/docs/site/src/content/docs/what-is-a-software-pack.md b/docs/site/src/content/docs/what-is-a-software-pack.md index cd40ebb..e7cac52 100644 --- a/docs/site/src/content/docs/what-is-a-software-pack.md +++ b/docs/site/src/content/docs/what-is-a-software-pack.md @@ -32,7 +32,7 @@ From that declaration, the Nebari Operator creates: - A **Certificate** for HTTPS connections (via cert-manager) - A **Keycloak client** for user authentication (via an Envoy Gateway SecurityPolicy) -See the [NebariApp CRD reference](/nebariapp-crd-reference/) for the full field list, and [Authentication flow](/auth-flow/) for how the OIDC login sequence works end-to-end. +See the [generated field reference](https://github.com/nebari-dev/nebari-operator/blob/v0.1.1/docs/api-reference.md) for every field, and [Authentication flow](/auth-flow/) for how the OIDC login sequence works end-to-end. ## Pack lifecycle From 788ea785e07ea155b842303d6fb797fbb7cd95e3 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 8 Oct 2026 10:30:29 +0200 Subject: [PATCH 13/16] Keep the clientSecretRef, Ready and displayName caveats Retiring the field tables dropped three corrections the table rows carried and the generated reference does not: spec.auth.clientSecretRef is ignored and the Secret is always -oidc-client (nebari-operator#193), Ready=True does not wait for RoutingReady, TLSReady or AuthReady, and landingPage.displayName is not validated. Add them to "What the generated reference does not tell you" in both copies. Each was checked against the v0.1.1 source. --- docs/nebariapp-crd-reference.md | 11 +++++++++++ docs/site/src/content/docs/nebariapp-crd-reference.md | 11 +++++++++++ 2 files changed, 22 insertions(+) diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index 0e02eb5..b1f02b3 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -32,6 +32,17 @@ behind the types the way this page did. realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](auth-flow.md#reading-user-identity-in-your-app)). +- **`auth.clientSecretRef` is ignored in v0.1.1.** The operator always reads and writes the + Secret named `-oidc-client`, with keys `client-id`, `client-secret`, and + `issuer-url`. If you manage credentials yourself (`provisionClient: false`), create the + Secret under that name. `status.clientSecretRef` is not written either + ([nebari-operator#193](https://github.com/nebari-dev/nebari-operator/issues/193)). +- **`Ready=True` does not mean the app is reachable.** `Ready` reflects the core checks + (namespace label, Service, validation) and hard reconcile failures. It does not wait for + `RoutingReady`, `TLSReady`, or `AuthReady`: a NebariApp with `TLSReady=False` still reports + `Ready=True`. Check the conditions you depend on. +- **`landingPage.displayName` is not validated.** Set it whenever `landingPage.enabled` is + `true`. The operator does not reject a missing value. - **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` and `landingPage.iconDark` were added after alpha.20 and are not available there. diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 3e3e96c..089d527 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -34,6 +34,17 @@ behind the types the way this page did. realm reaches the app ([nebari-operator#153](https://github.com/nebari-dev/nebari-operator/issues/153)). To restrict access by group today, verify the token in your app and check its `groups` claim (see [Authentication Flow](/auth-flow/#reading-user-identity-in-your-app)). +- **`auth.clientSecretRef` is ignored in v0.1.1.** The operator always reads and writes the + Secret named `-oidc-client`, with keys `client-id`, `client-secret`, and + `issuer-url`. If you manage credentials yourself (`provisionClient: false`), create the + Secret under that name. `status.clientSecretRef` is not written either + ([nebari-operator#193](https://github.com/nebari-dev/nebari-operator/issues/193)). +- **`Ready=True` does not mean the app is reachable.** `Ready` reflects the core checks + (namespace label, Service, validation) and hard reconcile failures. It does not wait for + `RoutingReady`, `TLSReady`, or `AuthReady`: a NebariApp with `TLSReady=False` still reports + `Ready=True`. Check the conditions you depend on. +- **`landingPage.displayName` is not validated.** Set it whenever `landingPage.enabled` is + `true`. The operator does not reject a missing value. - **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` and `landingPage.iconDark` were added after alpha.20 and are not available there. From 4718fed3efb483d9b7efe0aa90cc38e78fdf9da4 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 8 Oct 2026 11:22:08 +0200 Subject: [PATCH 14/16] Address review findings on token verification and docs accuracy Docs: - README's "key code" snippet now matches main.py, including options={"require": ["exp", "iss", "aud"]}; without it PyJWT accepts a token with no exp. - Raise the documented PyJWT floor to 2.10.1. 2.10.0 does a partial match on the issuer (CVE-2024-53861). - README no longer says NIC sets KEYCLOAK_ISSUER_SERVICE_PORT. NIC sets the context path and serves Keycloak on the default port, 8080. - CRD reference: a hand-managed client must be named -, because the SecurityPolicy always uses that client ID. Link the Ready and displayName caveats to nebari-operator#195 and #196. - auth-flow: serviceAccountName defaults to the NebariApp name; issuer-url is always written but may be empty; local login needs make update-hosts for keycloak.nebari.local; link the trust-model question to nebari-operator#194. The snippet drops cache_keys and logs JWKS outages, matching main.py. - Stop naming the stale operator version in the CRD page's history. Example: - Drop cache_keys=True, whose per-kid cache never expires, so a key removed from Keycloak's JWKS stops being trusted within the 5 minute JWK-set cache. - Log a JWKS connection failure at warning instead of as a rejected token. Requests still fail closed. - Test alg=none and HS256-keyed-with-the-public-key tokens, and the JWKS outage path. Widening algorithms now fails the HS256 case. - NetworkPolicy comment says the proxy selector is platform-owned and links nebari-operator#197. CI: - Lint fails when an operator docs or release link differs from OPERATOR_REF, or when prose names an operator version the integration matrix does not test. - Pin kubeconform to v0.8.0 and check its SHA-256. - Retry the forged-token probe while the deleted NetworkPolicy is still being enforced. --- .github/workflows/lint.yaml | 30 +++++++++++++- .github/workflows/test-integration.yaml | 7 +++- README.md | 5 ++- docs/auth-flow.md | 26 +++++++++--- docs/nebariapp-crd-reference.md | 16 +++++--- docs/site/src/content/docs/auth-flow.md | 26 +++++++++--- .../content/docs/nebariapp-crd-reference.md | 16 +++++--- examples/auth-fastapi/app/main.py | 7 +++- .../chart/templates/networkpolicy.yaml | 5 +++ examples/auth-fastapi/tests/test_main.py | 40 +++++++++++++++++++ 10 files changed, 149 insertions(+), 29 deletions(-) diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 2c45d46..99315da 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -17,8 +17,12 @@ jobs: - name: Install kubeconform run: | - curl -sSL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | \ - sudo tar -xz -C /usr/local/bin kubeconform + KUBECONFORM_VERSION=v0.8.0 + KUBECONFORM_SHA256=9bc2bffbf71f261128533edaf912153948b7ff238f9a531ae6d34466ec287883 + curl -sSLo /tmp/kubeconform.tar.gz \ + "https://github.com/yannh/kubeconform/releases/download/${KUBECONFORM_VERSION}/kubeconform-linux-amd64.tar.gz" + echo "${KUBECONFORM_SHA256} /tmp/kubeconform.tar.gz" | sha256sum -c - + sudo tar -xzf /tmp/kubeconform.tar.gz -C /usr/local/bin kubeconform # Vanilla YAML example - name: Validate vanilla-yaml manifests @@ -98,6 +102,28 @@ jobs: echo "ArgoCD Application examples must use 'project: nebari-apps'"; exit 1 fi + - name: Operator version references match OPERATOR_REF + run: | + # dev/Makefile's OPERATOR_REF is the one pinned operator version. Links to + # operator docs and releases must use it, and prose that names an operator + # version must name one the integration matrix tests. + set -euo pipefail + ref=$(sed -n 's/^OPERATOR_REF ?= //p' dev/Makefile) + tested=$(yq '.jobs.*.strategy.matrix.operator-ref[]' .github/workflows/test-integration.yaml | sort -u) + echo "OPERATOR_REF=$ref; CI matrix: $(echo $tested)" + echo "$tested" | grep -qx "$ref" || { echo "OPERATOR_REF $ref is not in the integration matrix"; exit 1; } + fail=0 + # Operator doc and release links must point at the pinned version. + while IFS= read -r hit; do + echo "::error::$hit links a different operator version than OPERATOR_REF ($ref)"; fail=1 + done < <(git grep -nE 'nebari-operator/(blob|tree|releases/download)/v[0-9]' -- '*.md' '*.yaml' '*.yml' | grep -v "nebari-operator/[a-z/]*/$ref/" || true) + # Prose naming an operator version must name one CI actually tests. + while IFS= read -r hit; do + v=$(echo "$hit" | grep -oE 'operator `?v[0-9]+\.[0-9]+\.[0-9]+[-a-z0-9.]*' | head -1 | grep -oE 'v[0-9].*' | sed 's/[.]$//') + echo "$tested" | grep -qx "$v" || { echo "::error::$hit names operator $v, which the integration matrix does not test"; fail=1; } + done < <(git grep -nE 'operator `?v[0-9]+\.[0-9]+\.[0-9]+' -- '*.md' || true) + exit $fail + app-tests: runs-on: ubuntu-latest steps: diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index 5e9b677..88e0b97 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -213,7 +213,12 @@ jobs: out=$(probe || true) echo "$out" | grep -q 'curl_exit=28' || { echo "NetworkPolicy did not block direct access"; echo "$out"; exit 1; } kubectl delete networkpolicy test-fastapi-my-pack --wait - out=$(probe || true) + # The CNI can take a few seconds to drop the rule after the delete. + for attempt in 1 2 3 4 5 6; do + out=$(probe || true) + echo "$out" | grep -q 'curl_exit=28' || break + echo "probe $attempt still blocked, retrying"; sleep 5 + done echo "$out" | grep -q 'could not be verified' || { echo "forged token was not rejected"; echo "$out"; exit 1; } if echo "$out" | grep -q mallory; then echo "forged identity was displayed"; exit 1; fi echo "forged token rejected" diff --git a/README.md b/README.md index 972d974..dd3f9f5 100644 --- a/README.md +++ b/README.md @@ -436,6 +436,8 @@ claims = jwt.decode( algorithms=["RS256"], audience=CLIENT_ID, # client-id from -oidc-client issuer=ISSUER, # issuer-url from -oidc-client + # PyJWT accepts a token with no exp unless told to require it. + options={"require": ["exp", "iss", "aud"]}, ) ``` @@ -960,7 +962,8 @@ kubectl get securitypolicy my-pack-security -n my-pack \ `OIDC: error fetching endpoints from issuer` means the operator is pointing Envoy Gateway at the wrong in-cluster Keycloak URL. Set `KEYCLOAK_ISSUER_SERVICE_PORT` and `KEYCLOAK_ISSUER_CONTEXT_PATH` on the operator Deployment to match your Keycloak -Service (NIC sets these; `dev/configure-operator.sh` shows the kind equivalent). +Service. NIC sets the context path and runs Keycloak on the operator's default port, +8080; `dev/configure-operator.sh` shows the kind equivalent. ### TLS certificate not provisioning diff --git a/docs/auth-flow.md b/docs/auth-flow.md index b26bdcd..b7d1f32 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -132,6 +132,9 @@ in it.** Envoy Gateway does not do this for you: in `routing.publicRoutes` are served by an HTTPRoute with no SecurityPolicy attached. A request on any of those paths can carry any `IdToken-*` cookie it likes. +Whether the platform should state or enforce this itself is tracked in +[nebari-operator#194](https://github.com/nebari-dev/nebari-operator/issues/194). + Verification needs three values, all available to your pod: | Value | Where it comes from | @@ -140,12 +143,19 @@ Verification needs three values, all available to your pod: | Issuer (the token's `iss`) | `issuer-url` key of the same Secret. It is empty unless the operator runs with `KEYCLOAK_EXTERNAL_URL`; in that case use the issuer your Keycloak puts in tokens. | | JWKS URL | Keycloak serves it at `/protocol/openid-connect/certs`. Use the in-cluster Keycloak URL if your pods cannot reach the public one. | -With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10`): +With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10.1`; 2.10.0 has a broken issuer check, +[CVE-2024-53861](https://github.com/advisories/GHSA-75c5-xw7c-p5pm)): ```python +import logging + import jwt -jwks = jwt.PyJWKClient(JWKS_URL, cache_keys=True) +log = logging.getLogger(__name__) + +# The JWK set is cached for 5 minutes. Avoid cache_keys=True: its per-key cache never +# expires, so a key Keycloak has removed would stay trusted until the process restarts. +jwks = jwt.PyJWKClient(JWKS_URL, timeout=5) def verified_claims(request) -> dict | None: @@ -162,6 +172,9 @@ def verified_claims(request) -> dict | None: issuer=ISSUER, options={"require": ["exp", "iss", "aud"]}, ) + except jwt.PyJWKClientConnectionError: + log.warning("JWKS unreachable; treating request as unauthenticated") + return None except jwt.PyJWTError: return None ``` @@ -227,8 +240,8 @@ data: device-client-id: # Present when deviceFlowClient is enabled. ``` -The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` `get` -this Secret through the Kubernetes API: +The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` (default: the +NebariApp name) `get` this Secret through the Kubernetes API: - **Role:** `-oidc-secret-reader` - **RoleBinding:** `-oidc-secret-reader` @@ -375,7 +388,7 @@ extraEnvRaw: secretKeyRef: name: -oidc-client key: issuer-url - optional: true # May not be present in all configurations + optional: true # Always written, but empty unless the operator sets KEYCLOAK_EXTERNAL_URL ``` The OIDC discovery URL can be constructed as: @@ -438,7 +451,8 @@ for the full set of OIDC fields. - **Local development:** The `dev/` Makefile builds a kind cluster with Keycloak, Envoy Gateway, cert-manager and the operator, so you can test the full login flow locally: - `cd dev && make up-fastapi`, then log in with the Keycloak credentials it prints. The + `cd dev && make up-fastapi`, then `make update-hosts` so the browser can resolve + `keycloak.nebari.local`, then log in with the Keycloak credentials it prints. The FastAPI example shows "Not Authenticated" when no valid IdToken cookie is present. - **Token expiration:** Envoy Gateway handles token refresh automatically via refresh diff --git a/docs/nebariapp-crd-reference.md b/docs/nebariapp-crd-reference.md index b1f02b3..72c7fc3 100644 --- a/docs/nebariapp-crd-reference.md +++ b/docs/nebariapp-crd-reference.md @@ -8,7 +8,7 @@ Kustomize, or Helm. **Operator version this page tracks:** `v0.1.1` This page used to hold a hand-maintained copy of the NebariApp field reference. It went stale: -it tracked operator `v0.1.0-alpha.19` while the operator moved on, and nothing detected the +it tracked an operator release from several versions back, and nothing detected the drift. Two copies of it lived in this repository, and the operator generates a third from the Go types. The field reference now has one home, next to the code that enforces it. @@ -35,14 +35,18 @@ behind the types the way this page did. - **`auth.clientSecretRef` is ignored in v0.1.1.** The operator always reads and writes the Secret named `-oidc-client`, with keys `client-id`, `client-secret`, and `issuer-url`. If you manage credentials yourself (`provisionClient: false`), create the - Secret under that name. `status.clientSecretRef` is not written either + Secret under that name, and name the client in your identity provider + `-`: the gateway's SecurityPolicy always uses that client ID, + not the Secret's `client-id`. `status.clientSecretRef` is not written either ([nebari-operator#193](https://github.com/nebari-dev/nebari-operator/issues/193)). - **`Ready=True` does not mean the app is reachable.** `Ready` reflects the core checks (namespace label, Service, validation) and hard reconcile failures. It does not wait for `RoutingReady`, `TLSReady`, or `AuthReady`: a NebariApp with `TLSReady=False` still reports - `Ready=True`. Check the conditions you depend on. + `Ready=True`. Check the conditions you depend on + ([nebari-operator#195](https://github.com/nebari-dev/nebari-operator/issues/195)). - **`landingPage.displayName` is not validated.** Set it whenever `landingPage.enabled` is - `true`. The operator does not reject a missing value. + `true`. The operator does not reject a missing value + ([nebari-operator#196](https://github.com/nebari-dev/nebari-operator/issues/196)). - **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` and `landingPage.iconDark` were added after alpha.20 and are not available there. @@ -92,8 +96,8 @@ creates. If the namespace already exists, label it yourself. ## Who can read the OIDC Secret When `provisionClient` is true, the operator writes `-oidc-client` and creates a Role -(`-oidc-secret-reader`) that lets `spec.serviceAccountName` `get` that Secret through the -Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from +(`-oidc-secret-reader`) that lets `spec.serviceAccountName` (default: the NebariApp name) `get` that Secret +through the Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from anyone else. Kubernetes RBAC is additive, so these can also read the Secret: - anyone who already has `get` on Secrets in the namespace, such as namespace admins diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index 04705f6..4911395 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -134,6 +134,9 @@ in it.** Envoy Gateway does not do this for you: in `routing.publicRoutes` are served by an HTTPRoute with no SecurityPolicy attached. A request on any of those paths can carry any `IdToken-*` cookie it likes. +Whether the platform should state or enforce this itself is tracked in +[nebari-operator#194](https://github.com/nebari-dev/nebari-operator/issues/194). + Verification needs three values, all available to your pod: | Value | Where it comes from | @@ -142,12 +145,19 @@ Verification needs three values, all available to your pod: | Issuer (the token's `iss`) | `issuer-url` key of the same Secret. It is empty unless the operator runs with `KEYCLOAK_EXTERNAL_URL`; in that case use the issuer your Keycloak puts in tokens. | | JWKS URL | Keycloak serves it at `/protocol/openid-connect/certs`. Use the in-cluster Keycloak URL if your pods cannot reach the public one. | -With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10`): +With [PyJWT](https://pyjwt.readthedocs.io/) (`PyJWT[crypto]>=2.10.1`; 2.10.0 has a broken issuer check, +[CVE-2024-53861](https://github.com/advisories/GHSA-75c5-xw7c-p5pm)): ```python +import logging + import jwt -jwks = jwt.PyJWKClient(JWKS_URL, cache_keys=True) +log = logging.getLogger(__name__) + +# The JWK set is cached for 5 minutes. Avoid cache_keys=True: its per-key cache never +# expires, so a key Keycloak has removed would stay trusted until the process restarts. +jwks = jwt.PyJWKClient(JWKS_URL, timeout=5) def verified_claims(request) -> dict | None: @@ -164,6 +174,9 @@ def verified_claims(request) -> dict | None: issuer=ISSUER, options={"require": ["exp", "iss", "aud"]}, ) + except jwt.PyJWKClientConnectionError: + log.warning("JWKS unreachable; treating request as unauthenticated") + return None except jwt.PyJWTError: return None ``` @@ -229,8 +242,8 @@ data: device-client-id: # Present when deviceFlowClient is enabled. ``` -The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` `get` -this Secret through the Kubernetes API: +The operator also creates a Role and RoleBinding that let `spec.serviceAccountName` (default: the +NebariApp name) `get` this Secret through the Kubernetes API: - **Role:** `-oidc-secret-reader` - **RoleBinding:** `-oidc-secret-reader` @@ -377,7 +390,7 @@ extraEnvRaw: secretKeyRef: name: -oidc-client key: issuer-url - optional: true # May not be present in all configurations + optional: true # Always written, but empty unless the operator sets KEYCLOAK_EXTERNAL_URL ``` The OIDC discovery URL can be constructed as: @@ -425,7 +438,8 @@ for the full set of OIDC fields. - **Local development:** The `dev/` Makefile builds a kind cluster with Keycloak, Envoy Gateway, cert-manager and the operator, so you can test the full login flow locally: - `cd dev && make up-fastapi`, then log in with the Keycloak credentials it prints. The + `cd dev && make up-fastapi`, then `make update-hosts` so the browser can resolve + `keycloak.nebari.local`, then log in with the Keycloak credentials it prints. The FastAPI example shows "Not Authenticated" when no valid IdToken cookie is present. - **Token expiration:** Envoy Gateway handles token refresh automatically via refresh diff --git a/docs/site/src/content/docs/nebariapp-crd-reference.md b/docs/site/src/content/docs/nebariapp-crd-reference.md index 089d527..5564e62 100644 --- a/docs/site/src/content/docs/nebariapp-crd-reference.md +++ b/docs/site/src/content/docs/nebariapp-crd-reference.md @@ -10,7 +10,7 @@ Kustomize, or Helm. **Operator version this page tracks:** `v0.1.1` This page used to hold a hand-maintained copy of the NebariApp field reference. It went stale: -it tracked operator `v0.1.0-alpha.19` while the operator moved on, and nothing detected the +it tracked an operator release from several versions back, and nothing detected the drift. Two copies of it lived in this repository, and the operator generates a third from the Go types. The field reference now has one home, next to the code that enforces it. @@ -37,14 +37,18 @@ behind the types the way this page did. - **`auth.clientSecretRef` is ignored in v0.1.1.** The operator always reads and writes the Secret named `-oidc-client`, with keys `client-id`, `client-secret`, and `issuer-url`. If you manage credentials yourself (`provisionClient: false`), create the - Secret under that name. `status.clientSecretRef` is not written either + Secret under that name, and name the client in your identity provider + `-`: the gateway's SecurityPolicy always uses that client ID, + not the Secret's `client-id`. `status.clientSecretRef` is not written either ([nebari-operator#193](https://github.com/nebari-dev/nebari-operator/issues/193)). - **`Ready=True` does not mean the app is reachable.** `Ready` reflects the core checks (namespace label, Service, validation) and hard reconcile failures. It does not wait for `RoutingReady`, `TLSReady`, or `AuthReady`: a NebariApp with `TLSReady=False` still reports - `Ready=True`. Check the conditions you depend on. + `Ready=True`. Check the conditions you depend on + ([nebari-operator#195](https://github.com/nebari-dev/nebari-operator/issues/195)). - **`landingPage.displayName` is not validated.** Set it whenever `landingPage.enabled` is - `true`. The operator does not reject a missing value. + `true`. The operator does not reject a missing value + ([nebari-operator#196](https://github.com/nebari-dev/nebari-operator/issues/196)). - **NIC v0.14.0 deploys operator `v0.1.0-alpha.20`, not `v0.1.1`.** `landingPage.iconLight` and `landingPage.iconDark` were added after alpha.20 and are not available there. @@ -94,8 +98,8 @@ creates. If the namespace already exists, label it yourself. ## Who can read the OIDC Secret When `provisionClient` is true, the operator writes `-oidc-client` and creates a Role -(`-oidc-secret-reader`) that lets `spec.serviceAccountName` `get` that Secret through the -Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from +(`-oidc-secret-reader`) that lets `spec.serviceAccountName` (default: the NebariApp name) `get` that Secret +through the Kubernetes API. That Role adds access for one ServiceAccount. It does not remove access from anyone else. Kubernetes RBAC is additive, so these can also read the Secret: - anyone who already has `get` on Secrets in the namespace, such as namespace admins diff --git a/examples/auth-fastapi/app/main.py b/examples/auth-fastapi/app/main.py index 1077116..46f6f23 100644 --- a/examples/auth-fastapi/app/main.py +++ b/examples/auth-fastapi/app/main.py @@ -54,6 +54,11 @@ def verify(self, token: str) -> dict | None: # but accepts a token with no exp unless told to require it. options={"require": ["exp", "iss", "aud"]}, ) + except jwt.PyJWKClientConnectionError as exc: + # Keycloak's JWKS endpoint is unreachable. This is an outage, not a + # bad token, so make it visible; the request still fails closed. + log.warning("could not fetch signing keys to verify IdToken: %s", exc) + return None except jwt.PyJWTError as exc: log.info("rejected IdToken: %s", exc) return None @@ -72,7 +77,7 @@ def verifier_from_env() -> TokenVerifier | None: if not issuer or not client_id: return None jwks_url = os.environ.get("OIDC_JWKS_URL") or f"{issuer.rstrip('/')}/protocol/openid-connect/certs" - return TokenVerifier(issuer, client_id, jwt.PyJWKClient(jwks_url, cache_keys=True, timeout=5)) + return TokenVerifier(issuer, client_id, jwt.PyJWKClient(jwks_url, timeout=5)) verifier = verifier_from_env() diff --git a/examples/auth-fastapi/chart/templates/networkpolicy.yaml b/examples/auth-fastapi/chart/templates/networkpolicy.yaml index 25f8499..0c7accf 100644 --- a/examples/auth-fastapi/chart/templates/networkpolicy.yaml +++ b/examples/auth-fastapi/chart/templates/networkpolicy.yaml @@ -3,6 +3,11 @@ cannot bypass the OIDC filter by calling the Service from inside the cluster. This does not cover routing.publicRoutes, which go through the gateway without a SecurityPolicy; the app's own token verification handles those. + + The proxy labels and gatewayNamespace below describe how the platform deploys + Envoy Gateway, not anything this pack controls. If you copy this file, expect to + update them when the platform changes; nebari-operator#197 tracks giving the + selector one owner in the nebari-app chart. */}} {{- if .Values.networkPolicy.enabled }} apiVersion: networking.k8s.io/v1 diff --git a/examples/auth-fastapi/tests/test_main.py b/examples/auth-fastapi/tests/test_main.py index c2b3e8b..50d7613 100644 --- a/examples/auth-fastapi/tests/test_main.py +++ b/examples/auth-fastapi/tests/test_main.py @@ -5,10 +5,16 @@ pytest tests """ +import base64 +import hashlib +import hmac +import json +import logging import time import jwt import pytest +from cryptography.hazmat.primitives import serialization from cryptography.hazmat.primitives.asymmetric import rsa from fastapi.testclient import TestClient @@ -53,6 +59,23 @@ def make_token(key=TRUSTED, kid="trusted", drop=(), **overrides): return jwt.encode(claims, key, algorithm="RS256", headers={"kid": kid}) +def _b64(data: bytes) -> str: + return base64.urlsafe_b64encode(data).rstrip(b"=").decode() + + +def forge_token(alg, kid="trusted"): + """Build a token PyJWT refuses to encode: alg=none, or HS256 keyed with the + trusted RSA public key (the classic algorithm-confusion attack).""" + claims = {"iss": ISSUER, "aud": CLIENT_ID, "exp": int(time.time()) + 300, "preferred_username": "alice"} + signing_input = f"{_b64(json.dumps({'alg': alg, 'kid': kid}).encode())}.{_b64(json.dumps(claims).encode())}" + if alg == "none": + return signing_input + "." + public_pem = TRUSTED.public_key().public_bytes( + serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo + ) + return f"{signing_input}.{_b64(hmac.new(public_pem, signing_input.encode(), hashlib.sha256).digest())}" + + @pytest.fixture def client(monkeypatch): monkeypatch.setattr(main, "verifier", main.TokenVerifier(ISSUER, CLIENT_ID, FakeJWKClient())) @@ -71,6 +94,8 @@ def client(monkeypatch): ("expired", {"IdToken-a1b2c3d4": make_token(exp=int(time.time()) - 60)}, False), ("no expiry", {"IdToken-a1b2c3d4": make_token(drop=["exp"])}, False), ("unsigned payload", {"IdToken-a1b2c3d4": "e30.eyJwcmVmZXJyZWRfdXNlcm5hbWUiOiJldmUifQ."}, False), + ("alg none with a trusted kid", {"IdToken-a1b2c3d4": forge_token("none")}, False), + ("HS256 keyed with the public key", {"IdToken-a1b2c3d4": forge_token("HS256")}, False), ( "extra IdToken cookie", {"IdToken-a1b2c3d4": make_token(), "IdToken-00000000": make_token(key=UNTRUSTED)}, @@ -97,3 +122,18 @@ def test_verification_not_configured_fails_closed(monkeypatch): def test_health(): assert TestClient(main.app).get("/health").json() == {"status": "ok"} + + +class UnreachableJWKClient: + def get_signing_key_from_jwt(self, token): + raise jwt.PyJWKClientConnectionError("connection refused") + + +def test_jwks_outage_fails_closed_and_warns(monkeypatch, caplog): + monkeypatch.setattr(main, "verifier", main.TokenVerifier(ISSUER, CLIENT_ID, UnreachableJWKClient())) + client = TestClient(main.app) + client.cookies.set("IdToken-a1b2c3d4", make_token()) + with caplog.at_level(logging.INFO, logger=main.log.name): + body = client.get("/").text + assert "alice" not in body + assert any(r.levelno == logging.WARNING and "signing keys" in r.getMessage() for r in caplog.records) From 0f500e0763c5201a5b107bd96037b2f140b57628 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 8 Oct 2026 11:37:02 +0200 Subject: [PATCH 15/16] Tighten the RS256 pin test and the operator version guard - Assert directly that TokenVerifier.verify passes algorithms=["RS256"]. The HS256 case only failed under a widened pin because PyJWT raised a TypeError the handler doesn't catch; adding "none" or ES256 passed. Note that the alg=none and HS256 cases document PyJWT's own refusal. - Move the operator version guard to dev/check-operator-refs.sh and check every match rather than every line. It now also covers releases/tag links, the "Operator version this page tracks" pins, and prose such as "not enforced in v0.1.1". tree/main stays allowed for the separately versioned nebari-app chart links. Keep the kustomize comment's version on one line with its context so the guard sees it. - Say "NetworkPolicy still blocks the Service" when the forged-token probe never gets through, instead of "forged token was not rejected". - The issuer-url comment now says the operator writes it only when it provisions the client. --- .github/workflows/lint.yaml | 21 +-------- .github/workflows/test-integration.yaml | 3 ++ dev/check-operator-refs.sh | 44 +++++++++++++++++++ docs/auth-flow.md | 3 +- docs/site/src/content/docs/auth-flow.md | 3 +- examples/auth-fastapi/tests/test_main.py | 16 +++++++ .../overlays/production/nebariapp-patch.yaml | 4 +- 7 files changed, 70 insertions(+), 24 deletions(-) create mode 100755 dev/check-operator-refs.sh diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 99315da..c73c38f 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -103,26 +103,7 @@ jobs: fi - name: Operator version references match OPERATOR_REF - run: | - # dev/Makefile's OPERATOR_REF is the one pinned operator version. Links to - # operator docs and releases must use it, and prose that names an operator - # version must name one the integration matrix tests. - set -euo pipefail - ref=$(sed -n 's/^OPERATOR_REF ?= //p' dev/Makefile) - tested=$(yq '.jobs.*.strategy.matrix.operator-ref[]' .github/workflows/test-integration.yaml | sort -u) - echo "OPERATOR_REF=$ref; CI matrix: $(echo $tested)" - echo "$tested" | grep -qx "$ref" || { echo "OPERATOR_REF $ref is not in the integration matrix"; exit 1; } - fail=0 - # Operator doc and release links must point at the pinned version. - while IFS= read -r hit; do - echo "::error::$hit links a different operator version than OPERATOR_REF ($ref)"; fail=1 - done < <(git grep -nE 'nebari-operator/(blob|tree|releases/download)/v[0-9]' -- '*.md' '*.yaml' '*.yml' | grep -v "nebari-operator/[a-z/]*/$ref/" || true) - # Prose naming an operator version must name one CI actually tests. - while IFS= read -r hit; do - v=$(echo "$hit" | grep -oE 'operator `?v[0-9]+\.[0-9]+\.[0-9]+[-a-z0-9.]*' | head -1 | grep -oE 'v[0-9].*' | sed 's/[.]$//') - echo "$tested" | grep -qx "$v" || { echo "::error::$hit names operator $v, which the integration matrix does not test"; fail=1; } - done < <(git grep -nE 'operator `?v[0-9]+\.[0-9]+\.[0-9]+' -- '*.md' || true) - exit $fail + run: dev/check-operator-refs.sh app-tests: runs-on: ubuntu-latest diff --git a/.github/workflows/test-integration.yaml b/.github/workflows/test-integration.yaml index 88e0b97..ac6196f 100644 --- a/.github/workflows/test-integration.yaml +++ b/.github/workflows/test-integration.yaml @@ -219,6 +219,9 @@ jobs: echo "$out" | grep -q 'curl_exit=28' || break echo "probe $attempt still blocked, retrying"; sleep 5 done + if echo "$out" | grep -q 'curl_exit=28'; then + echo "NetworkPolicy still blocks the Service after it was deleted"; echo "$out"; exit 1 + fi echo "$out" | grep -q 'could not be verified' || { echo "forged token was not rejected"; echo "$out"; exit 1; } if echo "$out" | grep -q mallory; then echo "forged identity was displayed"; exit 1; fi echo "forged token rejected" diff --git a/dev/check-operator-refs.sh b/dev/check-operator-refs.sh new file mode 100755 index 0000000..6acdfb2 --- /dev/null +++ b/dev/check-operator-refs.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Fail when the docs and examples name an operator version that dev/Makefile's +# OPERATOR_REF and the CI integration matrix don't agree on. +# +# - Links into nebari-operator (blob, tree, releases/download, releases/tag) +# must use OPERATOR_REF. `main` is allowed: those links point at the +# nebari-app chart, which is versioned separately. +# - "Operator version this page tracks" pins must equal OPERATOR_REF. +# - Prose that names an operator version ("in v0.1.1", "operator v0.1.1", +# "operator version: `v0.1.1`") must name one the integration matrix tests. +# +# Every match on a line is checked, not just the first. Run from the repo root. +set -euo pipefail + +ref=$(sed -n 's/^OPERATOR_REF ?= //p' dev/Makefile) +tested=$(yq '.jobs.*.strategy.matrix.operator-ref[]' .github/workflows/test-integration.yaml | sort -u) +echo "OPERATOR_REF=$ref; integration matrix: $(echo $tested)" + +fail=0 +err() { echo "::error::$1"; fail=1; } + +[ -n "$ref" ] || err "could not read OPERATOR_REF from dev/Makefile" +grep -qxF "$ref" <<<"$tested" || err "OPERATOR_REF $ref is not in the integration matrix" + +files=(-- '*.md' '*.yaml' '*.yml') +version='v[0-9]+\.[0-9]+\.[0-9]+(-[a-z0-9.]*[a-z0-9])?' + +# git grep -o prints one line per match, so a correct link can't hide a wrong one. +while IFS= read -r hit; do + v=${hit##*/} + [ "$v" = "$ref" ] || [ "$v" = main ] || err "$hit: links operator $v, not OPERATOR_REF ($ref)" +done < <(git grep -noE "nebari-operator/(blob|tree|releases/download|releases/tag)/[A-Za-z0-9._-]+" "${files[@]}" || true) + +while IFS= read -r hit; do + v=$(grep -oE "$version" <<<"$hit") + [ "$v" = "$ref" ] || err "$hit: page pin is $v, not OPERATOR_REF ($ref)" +done < <(git grep -noE "Operator version this page tracks:\*\* \`$version" "${files[@]}" || true) + +while IFS= read -r hit; do + v=$(grep -oE "$version" <<<"$hit") + grep -qxF "$v" <<<"$tested" || err "$hit: names operator $v, which the integration matrix does not test" +done < <(git grep -noiE "(\bin|operator( version:)?) \`?$version" "${files[@]}" || true) + +exit $fail diff --git a/docs/auth-flow.md b/docs/auth-flow.md index b7d1f32..aac8986 100644 --- a/docs/auth-flow.md +++ b/docs/auth-flow.md @@ -388,7 +388,8 @@ extraEnvRaw: secretKeyRef: name: -oidc-client key: issuer-url - optional: true # Always written, but empty unless the operator sets KEYCLOAK_EXTERNAL_URL + optional: true # Written by the operator when it provisions the client (empty unless + # KEYCLOAK_EXTERNAL_URL is set); with provisionClient: false, you write it ``` The OIDC discovery URL can be constructed as: diff --git a/docs/site/src/content/docs/auth-flow.md b/docs/site/src/content/docs/auth-flow.md index 4911395..45010c1 100644 --- a/docs/site/src/content/docs/auth-flow.md +++ b/docs/site/src/content/docs/auth-flow.md @@ -390,7 +390,8 @@ extraEnvRaw: secretKeyRef: name: -oidc-client key: issuer-url - optional: true # Always written, but empty unless the operator sets KEYCLOAK_EXTERNAL_URL + optional: true # Written by the operator when it provisions the client (empty unless + # KEYCLOAK_EXTERNAL_URL is set); with provisionClient: false, you write it ``` The OIDC discovery URL can be constructed as: diff --git a/examples/auth-fastapi/tests/test_main.py b/examples/auth-fastapi/tests/test_main.py index 50d7613..50d35c4 100644 --- a/examples/auth-fastapi/tests/test_main.py +++ b/examples/auth-fastapi/tests/test_main.py @@ -94,6 +94,8 @@ def client(monkeypatch): ("expired", {"IdToken-a1b2c3d4": make_token(exp=int(time.time()) - 60)}, False), ("no expiry", {"IdToken-a1b2c3d4": make_token(drop=["exp"])}, False), ("unsigned payload", {"IdToken-a1b2c3d4": "e30.eyJwcmVmZXJyZWRfdXNlcm5hbWUiOiJldmUifQ."}, False), + # PyJWT itself refuses both of these with a key object, whatever algorithms + # are allowed; test_verify_pins_rs256 is what guards the pin. ("alg none with a trusted kid", {"IdToken-a1b2c3d4": forge_token("none")}, False), ("HS256 keyed with the public key", {"IdToken-a1b2c3d4": forge_token("HS256")}, False), ( @@ -111,6 +113,20 @@ def test_index_only_trusts_verified_tokens(client, name, cookies, authenticated) assert ("Not Authenticated" in body) is not authenticated, name +def test_verify_pins_rs256(monkeypatch): + seen = {} + real_decode = jwt.decode + + def spy(*args, **kwargs): + seen.update(kwargs) + return real_decode(*args, **kwargs) + + monkeypatch.setattr(main.jwt, "decode", spy) + claims = main.TokenVerifier(ISSUER, CLIENT_ID, FakeJWKClient()).verify(make_token()) + assert claims["preferred_username"] == "alice" + assert seen["algorithms"] == ["RS256"] + + def test_verification_not_configured_fails_closed(monkeypatch): monkeypatch.setattr(main, "verifier", None) client = TestClient(main.app) diff --git a/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml b/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml index b095aaa..0c0a014 100644 --- a/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml +++ b/examples/kustomize-nginx/overlays/production/nebariapp-patch.yaml @@ -6,8 +6,8 @@ spec: hostname: my-pack.nebari.example.com auth: enabled: true - # Creates these groups in Keycloak. NOT enforced at the gateway in operator - # v0.1.1: any user in the realm can log in (nebari-operator#153). + # Creates these groups in Keycloak. NOT enforced by the gateway in v0.1.1: + # any user in the realm can log in (nebari-operator#153). groups: - admin - data-science-team From 36849ad3c45b000750a94abafa0475b59fba9809 Mon Sep 17 00:00:00 2001 From: Chuck McAndrew <6248903+dcmcand@users.noreply.github.com> Date: Thu, 8 Oct 2026 13:01:48 +0200 Subject: [PATCH 16/16] Report uppercase operator versions and check the RS256 pin first --- dev/check-operator-refs.sh | 4 ++-- examples/auth-fastapi/tests/test_main.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dev/check-operator-refs.sh b/dev/check-operator-refs.sh index 6acdfb2..694c303 100755 --- a/dev/check-operator-refs.sh +++ b/dev/check-operator-refs.sh @@ -32,12 +32,12 @@ while IFS= read -r hit; do done < <(git grep -noE "nebari-operator/(blob|tree|releases/download|releases/tag)/[A-Za-z0-9._-]+" "${files[@]}" || true) while IFS= read -r hit; do - v=$(grep -oE "$version" <<<"$hit") + v=$(grep -oiE "$version" <<<"$hit") [ "$v" = "$ref" ] || err "$hit: page pin is $v, not OPERATOR_REF ($ref)" done < <(git grep -noE "Operator version this page tracks:\*\* \`$version" "${files[@]}" || true) while IFS= read -r hit; do - v=$(grep -oE "$version" <<<"$hit") + v=$(grep -oiE "$version" <<<"$hit") grep -qxF "$v" <<<"$tested" || err "$hit: names operator $v, which the integration matrix does not test" done < <(git grep -noiE "(\bin|operator( version:)?) \`?$version" "${files[@]}" || true) diff --git a/examples/auth-fastapi/tests/test_main.py b/examples/auth-fastapi/tests/test_main.py index 50d35c4..e2c73d7 100644 --- a/examples/auth-fastapi/tests/test_main.py +++ b/examples/auth-fastapi/tests/test_main.py @@ -123,8 +123,8 @@ def spy(*args, **kwargs): monkeypatch.setattr(main.jwt, "decode", spy) claims = main.TokenVerifier(ISSUER, CLIENT_ID, FakeJWKClient()).verify(make_token()) - assert claims["preferred_username"] == "alice" assert seen["algorithms"] == ["RS256"] + assert claims["preferred_username"] == "alice" def test_verification_not_configured_fails_closed(monkeypatch):