Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,7 @@ discovery endpoint or its TLS CA.
| podLabels | object | `{}` | Extra labels to add to the gateway pod. |
| podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. |
| podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. |
| priorityClassName | string | `""` | PriorityClass for the gateway pod. The gateway is a control-plane component; when it shares nodes with the workloads it serves, the default priority gives it no advantage under node pressure. Empty = omit the field. |
| probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. |
| probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. |
| probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. |
Expand Down Expand Up @@ -758,6 +759,7 @@ discovery endpoint or its TLS CA.
| supervisor.replicas | int | `1` | Shared supervisor pool members, used only when gatewayConfig selects supervisor_topology = "gateway". The gateway always coordinates the shared supervisor as a high-availability pool: when a member fails or leaves, another member takes over its sandboxes without restarting their workloads. 1 has no failover; use 2 or more to survive losing a member. Only the supervisor Deployment's replica count depends on this value, so changing it neither restarts the gateway nor replaces running members. Ignored when supervisor.autoscaling.enabled is true. |
| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. |
| tolerations | list | `[]` | Tolerations for the gateway pod. |
| topologySpreadConstraints | list | `[]` | Topology spread constraints for the gateway pod, used to spread replicas across zones or nodes. Empty = omit the field. |
| upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. |
| upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. |
| upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. |
Expand Down
7 changes: 7 additions & 0 deletions deploy/helm/openshell/templates/_gateway-workload.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,9 @@ spec:
{{- end }}
- name: gateway-state
emptyDir: {}
{{- with .Values.priorityClassName }}
priorityClassName: {{ . }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 4 }}
Expand All @@ -370,4 +373,8 @@ spec:
tolerations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
107 changes: 107 additions & 0 deletions deploy/helm/openshell/tests/gateway_scheduling_test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

suite: gateway scheduling fields
templates:
- templates/gateway-config.yaml
- templates/statefulset.yaml
- templates/deployment.yaml
release:
name: openshell
namespace: my-namespace

tests:
- it: omits scheduling overrides by default on the Deployment
template: templates/deployment.yaml
set:
workload.kind: deployment
server.externalDbSecret: openshell-db
asserts:
- notExists:
path: spec.template.spec.priorityClassName
- notExists:
path: spec.template.spec.topologySpreadConstraints

# priorityClassName — shared pod template, so both workload shapes
- it: omits priorityClassName by default on the StatefulSet
template: templates/statefulset.yaml
asserts:
- notExists:
path: spec.template.spec.priorityClassName

- it: sets priorityClassName on the StatefulSet when configured
template: templates/statefulset.yaml
set:
priorityClassName: system-cluster-critical
asserts:
- equal:
path: spec.template.spec.priorityClassName
value: system-cluster-critical
- contains:
path: spec.template.spec.volumes
content:
name: gateway-state
emptyDir: {}

- it: sets priorityClassName on the Deployment when configured
template: templates/deployment.yaml
set:
workload.kind: deployment
server.externalDbSecret: openshell-db
priorityClassName: system-cluster-critical
asserts:
- equal:
path: spec.template.spec.priorityClassName
value: system-cluster-critical

# topologySpreadConstraints — shared pod template, so both workload shapes
- it: omits topologySpreadConstraints by default on the StatefulSet
template: templates/statefulset.yaml
asserts:
- notExists:
path: spec.template.spec.topologySpreadConstraints

- it: sets topologySpreadConstraints on the StatefulSet when configured
template: templates/statefulset.yaml
set:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: openshell
app.kubernetes.io/instance: openshell
asserts:
- equal:
path: spec.template.spec.topologySpreadConstraints[0].maxSkew
value: 1
- equal:
path: spec.template.spec.topologySpreadConstraints[0].topologyKey
value: topology.kubernetes.io/zone
- equal:
path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable
value: DoNotSchedule
- equal:
path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels
value:
app.kubernetes.io/name: openshell
app.kubernetes.io/instance: openshell

- it: sets topologySpreadConstraints on the Deployment when configured
template: templates/deployment.yaml
set:
workload.kind: deployment
server.externalDbSecret: openshell-db
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: openshell
app.kubernetes.io/instance: openshell
asserts:
- equal:
path: spec.template.spec.topologySpreadConstraints[0].topologyKey
value: kubernetes.io/hostname
9 changes: 9 additions & 0 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,15 @@ tolerations: []
# -- Affinity rules for the gateway pod.
affinity: {}

# -- PriorityClass for the gateway pod. The gateway is a control-plane
# component; when it shares nodes with the workloads it serves, the default
# priority gives it no advantage under node pressure. Empty = omit the field.
priorityClassName: ""

# -- Topology spread constraints for the gateway pod, used to spread replicas
# across zones or nodes. Empty = omit the field.
topologySpreadConstraints: []

# -- Non-secret gateway application configuration. Top-level keys name TOML
# tables and are rendered into the mounted gateway.toml file. Kubernetes
# resource inputs remain outside this map; template expressions derive the
Expand Down
2 changes: 2 additions & 0 deletions docs/kubernetes/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,8 @@ The most commonly changed values are:
| `global.image.registry` / `global.image.tag` / `global.image.pullPolicy` | Shared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence. |
| `gateway.image.registry` / `gateway.image.repository` / `gateway.image.tag` / `gateway.image.digest` | Gateway container image. Defaults to `ghcr.io/nvidia/openshell/gateway:latest`. |
| `replicaCount` | Number of gateway replicas. Values above `1` require shared PostgreSQL through `server.externalDbSecret`. |
| `priorityClassName` | Existing Kubernetes PriorityClass for gateway pods. Empty by default. |
| `topologySpreadConstraints` | Kubernetes topology spread constraints for gateway replicas. Empty by default. Set selectors to match the gateway pod labels. |
| `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. |
| `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. |
| `autoscaling.enabled` / `autoscaling.minReplicas` / `autoscaling.maxReplicas` | Render a HorizontalPodAutoscaler instead of a fixed `replicaCount`. Refer to [High Availability](/kubernetes/high-availability#scale-the-gateway). |
Expand Down
8 changes: 8 additions & 0 deletions skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -530,6 +530,14 @@ kubectl -n openshell logs <gateway-pod> -c openshell-gateway --tail=200
`server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway
pod, and the gateway propagates the effective value to sandbox supervisors.

Gateway pod scheduling accepts top-level `nodeSelector`, `affinity`,
`tolerations`, `priorityClassName`, and `topologySpreadConstraints` values for
both StatefulSet and Deployment workloads. The last two default to unset.
If a gateway pod stays Pending, inspect its scheduling events, confirm the
named PriorityClass exists (`kubectl get priorityclass`), and check that topology
spread selectors match the gateway pod labels and that eligible nodes have the
required topology labels. These values apply only to gateway pods.

#### Additional destination CA ConfigMap

When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the
Expand Down
Loading