Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,16 @@ sandbox JWT signing Secret. This precedence applies even if
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.

Operator mTLS gives an operator application or user an administrative principal
for managing the OpenShell gateway across all workspaces. Use a client
certificate with exact `OU=operator` and client authentication usage, signed by
a CA trusted by the gateway's configured client CA bundle. Enable authentication through
`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true`.
See [mTLS operators](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/authentication#mtls-operators)
for authentication requirements, or
[Managing Certificates](https://docs.nvidia.com/openshell/latest/kubernetes/managing-certificates#issue-an-operator-client-certificate)
for an example using cert-manager to issue and renew the certificate.

## SPIFFE/SPIRE provider token grants

Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path`
Expand Down
10 changes: 10 additions & 0 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,16 @@ sandbox JWT signing Secret. This precedence applies even if
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.

Operator mTLS gives an operator application or user an administrative principal
for managing the OpenShell gateway across all workspaces. Use a client
certificate with exact `OU=operator` and client authentication usage, signed by
a CA trusted by the gateway's configured client CA bundle. Enable authentication through
`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true`.
See [mTLS operators](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/authentication#mtls-operators)
for authentication requirements, or
[Managing Certificates](https://docs.nvidia.com/openshell/latest/kubernetes/managing-certificates#issue-an-operator-client-certificate)
for an example using cert-manager to issue and renew the certificate.

## SPIFFE/SPIRE provider token grants

Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path`
Expand Down
10 changes: 10 additions & 0 deletions docs/how-it-works/gateways/authentication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,12 @@ The connection flow:

### mTLS Operators

An operator mTLS certificate authenticates an operator application or user as
an administrative principal for an OpenShell gateway. Use this identity for
gateway administration, including managing workspaces, sandboxes, and providers
across all workspaces. It supports both administrative automation and interactive
operator access.

Enable `[openshell.gateway.mtls_auth] operator_enabled = true` to recognize a
trusted client certificate whose organizational unit is exactly `operator`
(`OU=operator`). This creates a distinct operator principal with user and
Expand All @@ -95,6 +101,10 @@ setting.
See [Providers](/how-it-works/providers/overview#retrieve-runtime-credentials)
for the credential retrieval contract.

For a Helm deployment example using cert-manager and the chart's CA issuer,
see [Managing Certificates](/kubernetes/managing-certificates#issue-an-operator-client-certificate).
It covers how to issue an operator client certificate using cert-manager.

### Inspect the API with grpcurl

The primary gateway listener serves the gRPC reflection v1 protocol. Reflection
Expand Down
9 changes: 8 additions & 1 deletion docs/how-it-works/gateways/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -312,14 +312,21 @@ Set `[openshell.gateway.mtls_auth] enabled = true` to map a verified client cert

Set `[openshell.gateway.mtls_auth] operator_enabled = true`,
`--enable-operator-auth true`, or `OPENSHELL_ENABLE_OPERATOR_AUTH=true` to enable
direct mTLS operators. The default is false. TLS and a client CA are required.
an operator application or user to authenticate through direct mTLS as a gateway
administrative principal. The default is false. TLS and a client CA are required.
This setting is independent of `enabled` and OIDC. A verified certificate with
the exact `OU=operator` grants user/platform-admin capabilities across all
workspaces and access to `GetProviderCredentials`. Do not combine an operator
certificate with an `Authorization` header. See
[Authentication](/how-it-works/gateways/authentication#mtls-operators) for the
certificate trust boundary.

On Helm deployments, set
`gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true` and use an
operator client certificate signed by a trusted client CA.
See [Managing Certificates](/kubernetes/managing-certificates#issue-an-operator-client-certificate)
for an example using cert-manager to issue an operator client certificate.

The client-certificate handshake policy has no `require_client_auth` TOML field. Client certificates are optional at the transport layer; gateway RPC authorization enforces the configured user or sandbox identity.

`[openshell.gateway.tls]` supports optional SNI-based dual-certificate mode for deployments that need separate internal and external server certificates. Set `external_cert_path` and `external_key_path` to point at the external (e.g. ACME/publicly-trusted) certificate and key. List the hostnames that should be served with the external certificate in `external_server_names`. Connections whose TLS SNI hostname matches one of those names receive the external certificate; all other connections (including those with no SNI) receive the primary internal certificate from `cert_path`/`key_path`. Both fields must be set together — providing only one is a configuration error. On Kubernetes with the Helm chart, the external certificate is managed automatically when `certManager.serverIssuerRef.name` is set; the chart populates these fields from the cert-manager-issued external server certificate.
Expand Down
2 changes: 1 addition & 1 deletion docs/how-it-works/workspaces.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The CLI targets the `default` workspace unless you set `--workspace` or
separate from the `/sandbox` filesystem directory inside a sandbox.

For Kubernetes namespace mapping and managed per-workspace namespaces, see
[Workspace on Kubernetes](/kubernetes/workspaces).
[Workspaces on Kubernetes](/kubernetes/workspaces).

## Understand the Role Model

Expand Down
120 changes: 120 additions & 0 deletions docs/kubernetes/managing-certificates.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,126 @@ The chart also runs a pre-install hook in JWT-only mode to create the gateway's
sandbox JWT signing Secret. That Secret is separate from the cert-manager TLS
certificate Secrets and is mounted at `/etc/openshell-jwt`.

## Issue an operator client certificate

An operator client certificate gives an operator application or user an
administrative principal for managing an OpenShell gateway across all
workspaces. The following example issues that identity with a standalone
cert-manager `Certificate` managed outside the Helm release.

Save these Helm values as `operator-values.yaml`:

```yaml
certManager:
enabled: true
server:
disableTls: false
tls:
enableMtls: true

gatewayConfig:
openshell.gateway.mtls_auth:
operator_enabled: true
```

Install or upgrade using your deployment's other values files as well:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> --namespace openshell \
--values operator-values.yaml
```

The gateway must trust the CA issuing the client certificate. Keep the default
`certManager.clientCaFromServerTlsSecret: true` and a non-empty
`server.tls.clientCaSecretName`. Alternatively, set
`certManager.clientCaFromServerTlsSecret: false` and point
`server.tls.clientCaSecretName` at `certManager.caSecretName`.

Save this standalone manifest as `operator-certificate.yaml`. It assumes a
release named `openshell` in namespace `openshell`; adjust the namespace and
`issuerRef.name` to match the chart's `<fullname>-ca-issuer` Issuer.

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: openshell-operator-client
namespace: openshell
spec:
secretName: openshell-operator-client-tls
commonName: platform-operator
subject:
organizationalUnits:
- operator
duration: 168h
renewBefore: 24h
privateKey:
algorithm: ECDSA
size: 256
usages:
- client auth
- digital signature
issuerRef:
name: openshell-ca-issuer
kind: Issuer
group: cert-manager.io
```

Apply it and wait for issuance:

```shell
kubectl apply -f operator-certificate.yaml
kubectl -n openshell wait certificate/openshell-operator-client --for=condition=Ready --timeout=120s
```

The exact `OU=operator` grants gateway-wide administration and access to
`GetProviderCredentials` when operator authentication is enabled. Restrict
access to the dedicated Secret and permission to issue certificates under the
trusted client CA. See [mTLS operators](/how-it-works/gateways/authentication#mtls-operators)
for the authorization and trust boundary.

### Verify the certificate

If you have `kubectl` access and want to verify that the certificate
authenticates correctly, use the following example. Export the certificate,
key, and server trust bundle:

```shell
umask 077
mkdir -p operator-tls
kubectl -n openshell get secret openshell-operator-client-tls -o jsonpath='{.data.tls\.crt}' | base64 --decode > operator-tls/tls.crt
kubectl -n openshell get secret openshell-operator-client-tls -o jsonpath='{.data.tls\.key}' | base64 --decode > operator-tls/tls.key
kubectl -n openshell get secret openshell-server-tls -o jsonpath='{.data.ca\.crt}' | base64 --decode > operator-tls/ca.crt
```

Keep a direct Service port-forward running:

```shell
kubectl -n openshell port-forward svc/openshell 8090:8080
```

In another terminal, use `grpcurl` to check the authenticated identity:

```shell
grpcurl -cacert operator-tls/ca.crt \
-cert operator-tls/tls.crt -key operator-tls/tls.key \
-d '{}' localhost:8090 openshell.v1.OpenShell/GetCurrentUser
```

The response should identify `platform-operator`, include the `operator` role,
and report `identityProvider: "mtls"`. This confirms that the gateway recognizes
the certificate as an operator identity. Send requests without an
`Authorization` header, through a direct or TLS-passthrough connection.

If the gateway presents an external server certificate, use that issuer's server
trust bundle for `ca.crt` instead of the chart CA. The operator client
certificate remains signed by the chart CA in this example.

cert-manager renews the Kubernetes Secret according to the Certificate's
`duration` and `renewBefore`. Refresh any exported certificate and key files
after renewal and reconnect the application with the updated pair.

## Using a real Issuer for the server certificate

By default, cert-manager issues both the server and client certificates from
Expand Down
2 changes: 1 addition & 1 deletion docs/kubernetes/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ The chart automatically generates PKI secrets on first install using pre-install

The default Kubernetes driver puts all workspaces in one sandbox namespace. To
have the gateway create a namespace for each workspace, follow
[Workspace on Kubernetes](/kubernetes/workspaces).
[Workspaces on Kubernetes](/kubernetes/workspaces).

### Split gateway and workspace releases

Expand Down
2 changes: 1 addition & 1 deletion docs/kubernetes/workspaces.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Workspace on Kubernetes"
title: "Workspaces on Kubernetes"
sidebar-title: "Workspaces"
description: "Configure how OpenShell workspaces map to Kubernetes namespaces, including managed per-workspace namespaces."
keywords: "Kubernetes, Workspaces, Namespaces, Managed Mode, Helm, RBAC"
Expand Down
9 changes: 9 additions & 0 deletions skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -752,6 +752,15 @@ If the gateway pod is pending with `MountVolume.SetUp failed for volume
`templates/certgen.yaml` output and the hook Job logs; cert-manager creates TLS
Secrets but does not create the sandbox JWT signing Secret.

Operator certificates can be issued with a standalone cert-manager `Certificate`
using the chart's existing `<fullname>-ca-issuer`. Verify exact `OU=operator`,
client authentication usage, and a chain trusted by the gateway client CA.
Authorization needs `gatewayConfig["openshell.gateway.mtls_auth"].operator_enabled=true`
and direct mTLS. Check the standalone Certificate's Ready condition and its
Secret's subject and expiry. Keep this dedicated Secret outside gateway,
supervisor, and workload pods. Exported CLI certificate/key files need refreshing
after renewal. See Managing Certificates in the published Kubernetes docs.

If the gateway exits with `failed to read sandbox JWT signing key from
/etc/openshell-jwt/signing.pem`, verify that `openshell-jwt-keys` contains
`signing.pem`, `public.pem`, and `kid`, and that the gateway workload mounts the
Expand Down
Loading