diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index b43974dff1..14c76f5982 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -344,6 +344,16 @@ sandbox JWT signing Secret. This precedence applies even if external non-cert-manager TLS source manages TLS and you pre-create the sandbox JWT signing Secret. +Operator mTLS gives an operator application or user an administrative principal +for managing the OpenShell gateway across all workspaces. Use a client +certificate with exact `OU=operator` and client authentication usage, signed by +a CA trusted by the gateway's configured client CA bundle. Enable authentication through +`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true`. +See [mTLS operators](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/authentication#mtls-operators) +for authentication requirements, or +[Managing Certificates](https://docs.nvidia.com/openshell/latest/kubernetes/managing-certificates#issue-an-operator-client-certificate) +for an example using cert-manager to issue and renew the certificate. + ## SPIFFE/SPIRE provider token grants Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path` diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index a55cc57e81..12af689f9d 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -345,6 +345,16 @@ sandbox JWT signing Secret. This precedence applies even if external non-cert-manager TLS source manages TLS and you pre-create the sandbox JWT signing Secret. +Operator mTLS gives an operator application or user an administrative principal +for managing the OpenShell gateway across all workspaces. Use a client +certificate with exact `OU=operator` and client authentication usage, signed by +a CA trusted by the gateway's configured client CA bundle. Enable authentication through +`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true`. +See [mTLS operators](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/authentication#mtls-operators) +for authentication requirements, or +[Managing Certificates](https://docs.nvidia.com/openshell/latest/kubernetes/managing-certificates#issue-an-operator-client-certificate) +for an example using cert-manager to issue and renew the certificate. + ## SPIFFE/SPIRE provider token grants Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path` diff --git a/docs/how-it-works/gateways/authentication.mdx b/docs/how-it-works/gateways/authentication.mdx index b764d3fbfe..810dd550ae 100644 --- a/docs/how-it-works/gateways/authentication.mdx +++ b/docs/how-it-works/gateways/authentication.mdx @@ -71,6 +71,12 @@ The connection flow: ### mTLS Operators +An operator mTLS certificate authenticates an operator application or user as +an administrative principal for an OpenShell gateway. Use this identity for +gateway administration, including managing workspaces, sandboxes, and providers +across all workspaces. It supports both administrative automation and interactive +operator access. + Enable `[openshell.gateway.mtls_auth] operator_enabled = true` to recognize a trusted client certificate whose organizational unit is exactly `operator` (`OU=operator`). This creates a distinct operator principal with user and @@ -95,6 +101,10 @@ setting. See [Providers](/how-it-works/providers/overview#retrieve-runtime-credentials) for the credential retrieval contract. +For a Helm deployment example using cert-manager and the chart's CA issuer, +see [Managing Certificates](/kubernetes/managing-certificates#issue-an-operator-client-certificate). +It covers how to issue an operator client certificate using cert-manager. + ### Inspect the API with grpcurl The primary gateway listener serves the gRPC reflection v1 protocol. Reflection diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 4035a69eae..cde903efb2 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -312,7 +312,8 @@ Set `[openshell.gateway.mtls_auth] enabled = true` to map a verified client cert Set `[openshell.gateway.mtls_auth] operator_enabled = true`, `--enable-operator-auth true`, or `OPENSHELL_ENABLE_OPERATOR_AUTH=true` to enable -direct mTLS operators. The default is false. TLS and a client CA are required. +an operator application or user to authenticate through direct mTLS as a gateway +administrative principal. The default is false. TLS and a client CA are required. This setting is independent of `enabled` and OIDC. A verified certificate with the exact `OU=operator` grants user/platform-admin capabilities across all workspaces and access to `GetProviderCredentials`. Do not combine an operator @@ -320,6 +321,12 @@ certificate with an `Authorization` header. See [Authentication](/how-it-works/gateways/authentication#mtls-operators) for the certificate trust boundary. +On Helm deployments, set +`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true` and use an +operator client certificate signed by a trusted client CA. +See [Managing Certificates](/kubernetes/managing-certificates#issue-an-operator-client-certificate) +for an example using cert-manager to issue an operator client certificate. + The client-certificate handshake policy has no `require_client_auth` TOML field. Client certificates are optional at the transport layer; gateway RPC authorization enforces the configured user or sandbox identity. `[openshell.gateway.tls]` supports optional SNI-based dual-certificate mode for deployments that need separate internal and external server certificates. Set `external_cert_path` and `external_key_path` to point at the external (e.g. ACME/publicly-trusted) certificate and key. List the hostnames that should be served with the external certificate in `external_server_names`. Connections whose TLS SNI hostname matches one of those names receive the external certificate; all other connections (including those with no SNI) receive the primary internal certificate from `cert_path`/`key_path`. Both fields must be set together — providing only one is a configuration error. On Kubernetes with the Helm chart, the external certificate is managed automatically when `certManager.serverIssuerRef.name` is set; the chart populates these fields from the cert-manager-issued external server certificate. diff --git a/docs/how-it-works/workspaces.mdx b/docs/how-it-works/workspaces.mdx index df5e2e1486..b88224cc25 100644 --- a/docs/how-it-works/workspaces.mdx +++ b/docs/how-it-works/workspaces.mdx @@ -18,7 +18,7 @@ The CLI targets the `default` workspace unless you set `--workspace` or separate from the `/sandbox` filesystem directory inside a sandbox. For Kubernetes namespace mapping and managed per-workspace namespaces, see -[Workspace on Kubernetes](/kubernetes/workspaces). +[Workspaces on Kubernetes](/kubernetes/workspaces). ## Understand the Role Model diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index 59de870c81..8be188accb 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -60,6 +60,126 @@ The chart also runs a pre-install hook in JWT-only mode to create the gateway's sandbox JWT signing Secret. That Secret is separate from the cert-manager TLS certificate Secrets and is mounted at `/etc/openshell-jwt`. +## Issue an operator client certificate + +An operator client certificate gives an operator application or user an +administrative principal for managing an OpenShell gateway across all +workspaces. The following example issues that identity with a standalone +cert-manager `Certificate` managed outside the Helm release. + +Save these Helm values as `operator-values.yaml`: + +```yaml +certManager: + enabled: true +server: + disableTls: false + tls: + enableMtls: true + +gatewayConfig: + openshell.gateway.mtls_auth: + operator_enabled: true +``` + +Install or upgrade using your deployment's other values files as well: + +```shell +helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --version --namespace openshell \ + --values operator-values.yaml +``` + +The gateway must trust the CA issuing the client certificate. Keep the default +`certManager.clientCaFromServerTlsSecret: true` and a non-empty +`server.tls.clientCaSecretName`. Alternatively, set +`certManager.clientCaFromServerTlsSecret: false` and point +`server.tls.clientCaSecretName` at `certManager.caSecretName`. + +Save this standalone manifest as `operator-certificate.yaml`. It assumes a +release named `openshell` in namespace `openshell`; adjust the namespace and +`issuerRef.name` to match the chart's `-ca-issuer` Issuer. + +```yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: openshell-operator-client + namespace: openshell +spec: + secretName: openshell-operator-client-tls + commonName: platform-operator + subject: + organizationalUnits: + - operator + duration: 168h + renewBefore: 24h + privateKey: + algorithm: ECDSA + size: 256 + usages: + - client auth + - digital signature + issuerRef: + name: openshell-ca-issuer + kind: Issuer + group: cert-manager.io +``` + +Apply it and wait for issuance: + +```shell +kubectl apply -f operator-certificate.yaml +kubectl -n openshell wait certificate/openshell-operator-client --for=condition=Ready --timeout=120s +``` + +The exact `OU=operator` grants gateway-wide administration and access to +`GetProviderCredentials` when operator authentication is enabled. Restrict +access to the dedicated Secret and permission to issue certificates under the +trusted client CA. See [mTLS operators](/how-it-works/gateways/authentication#mtls-operators) +for the authorization and trust boundary. + +### Verify the certificate + +If you have `kubectl` access and want to verify that the certificate +authenticates correctly, use the following example. Export the certificate, +key, and server trust bundle: + +```shell +umask 077 +mkdir -p operator-tls +kubectl -n openshell get secret openshell-operator-client-tls -o jsonpath='{.data.tls\.crt}' | base64 --decode > operator-tls/tls.crt +kubectl -n openshell get secret openshell-operator-client-tls -o jsonpath='{.data.tls\.key}' | base64 --decode > operator-tls/tls.key +kubectl -n openshell get secret openshell-server-tls -o jsonpath='{.data.ca\.crt}' | base64 --decode > operator-tls/ca.crt +``` + +Keep a direct Service port-forward running: + +```shell +kubectl -n openshell port-forward svc/openshell 8090:8080 +``` + +In another terminal, use `grpcurl` to check the authenticated identity: + +```shell +grpcurl -cacert operator-tls/ca.crt \ + -cert operator-tls/tls.crt -key operator-tls/tls.key \ + -d '{}' localhost:8090 openshell.v1.OpenShell/GetCurrentUser +``` + +The response should identify `platform-operator`, include the `operator` role, +and report `identityProvider: "mtls"`. This confirms that the gateway recognizes +the certificate as an operator identity. Send requests without an +`Authorization` header, through a direct or TLS-passthrough connection. + +If the gateway presents an external server certificate, use that issuer's server +trust bundle for `ca.crt` instead of the chart CA. The operator client +certificate remains signed by the chart CA in this example. + +cert-manager renews the Kubernetes Secret according to the Certificate's +`duration` and `renewBefore`. Refresh any exported certificate and key files +after renewal and reconnect the application with the updated pair. + ## Using a real Issuer for the server certificate By default, cert-manager issues both the server and client certificates from diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index 6c40df47be..67c399b1b4 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -120,7 +120,7 @@ The chart automatically generates PKI secrets on first install using pre-install The default Kubernetes driver puts all workspaces in one sandbox namespace. To have the gateway create a namespace for each workspace, follow -[Workspace on Kubernetes](/kubernetes/workspaces). +[Workspaces on Kubernetes](/kubernetes/workspaces). ### Split gateway and workspace releases diff --git a/docs/kubernetes/workspaces.mdx b/docs/kubernetes/workspaces.mdx index 2683b47d21..95346a9818 100644 --- a/docs/kubernetes/workspaces.mdx +++ b/docs/kubernetes/workspaces.mdx @@ -1,7 +1,7 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Workspace on Kubernetes" +title: "Workspaces on Kubernetes" sidebar-title: "Workspaces" description: "Configure how OpenShell workspaces map to Kubernetes namespaces, including managed per-workspace namespaces." keywords: "Kubernetes, Workspaces, Namespaces, Managed Mode, Helm, RBAC" diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 1d13def5ed..9fc07bb011 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -752,6 +752,15 @@ If the gateway pod is pending with `MountVolume.SetUp failed for volume `templates/certgen.yaml` output and the hook Job logs; cert-manager creates TLS Secrets but does not create the sandbox JWT signing Secret. +Operator certificates can be issued with a standalone cert-manager `Certificate` +using the chart's existing `-ca-issuer`. Verify exact `OU=operator`, +client authentication usage, and a chain trusted by the gateway client CA. +Authorization needs `gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true` +and direct mTLS. Check the standalone Certificate's Ready condition and its +Secret's subject and expiry. Keep this dedicated Secret outside gateway, +supervisor, and workload pods. Exported CLI certificate/key files need refreshing +after renewal. See Managing Certificates in the published Kubernetes docs. + If the gateway exits with `failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pem`, verify that `openshell-jwt-keys` contains `signing.pem`, `public.pem`, and `kid`, and that the gateway workload mounts the