diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 58aa132eb6..bc6676b9f1 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -615,6 +615,7 @@ discovery endpoint or its TLS CA. | podLabels | object | `{}` | Extra labels to add to the gateway pod. | | podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. | | podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. | +| priorityClassName | string | `""` | PriorityClass for the gateway pod. The gateway is a control-plane component; when it shares nodes with the workloads it serves, the default priority gives it no advantage under node pressure. Empty = omit the field. | | probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. | | probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. | | probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. | @@ -758,6 +759,7 @@ discovery endpoint or its TLS CA. | supervisor.replicas | int | `1` | Shared supervisor pool members, used only when gatewayConfig selects supervisor_topology = "gateway". The gateway always coordinates the shared supervisor as a high-availability pool: when a member fails or leaves, another member takes over its sandboxes without restarting their workloads. 1 has no failover; use 2 or more to survive losing a member. Only the supervisor Deployment's replica count depends on this value, so changing it neither restarts the gateway nor replaces running members. Ignored when supervisor.autoscaling.enabled is true. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | | tolerations | list | `[]` | Tolerations for the gateway pod. | +| topologySpreadConstraints | list | `[]` | Topology spread constraints for the gateway pod, used to spread replicas across zones or nodes. Empty = omit the field. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | | upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. | diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 6b73241e6c..b0814a40b8 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -358,6 +358,9 @@ spec: {{- end }} - name: gateway-state emptyDir: {} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 4 }} @@ -370,4 +373,8 @@ spec: tolerations: {{- toYaml . | nindent 4 }} {{- end }} + {{- with .Values.topologySpreadConstraints }} + topologySpreadConstraints: + {{- toYaml . | nindent 4 }} + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/tests/gateway_scheduling_test.yaml b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml new file mode 100644 index 0000000000..3a1a755d02 --- /dev/null +++ b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml @@ -0,0 +1,107 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway scheduling fields +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml + - templates/deployment.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: omits scheduling overrides by default on the Deployment + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + asserts: + - notExists: + path: spec.template.spec.priorityClassName + - notExists: + path: spec.template.spec.topologySpreadConstraints + + # priorityClassName — shared pod template, so both workload shapes + - it: omits priorityClassName by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.priorityClassName + + - it: sets priorityClassName on the StatefulSet when configured + template: templates/statefulset.yaml + set: + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} + + - it: sets priorityClassName on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + + # topologySpreadConstraints — shared pod template, so both workload shapes + - it: omits topologySpreadConstraints by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.topologySpreadConstraints + + - it: sets topologySpreadConstraints on the StatefulSet when configured + template: templates/statefulset.yaml + set: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].maxSkew + value: 1 + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: topology.kubernetes.io/zone + - equal: + path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable + value: DoNotSchedule + - equal: + path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels + value: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + + - it: sets topologySpreadConstraints on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: kubernetes.io/hostname diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 0bbb2cd6b2..987744afbb 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -389,6 +389,15 @@ tolerations: [] # -- Affinity rules for the gateway pod. affinity: {} +# -- PriorityClass for the gateway pod. The gateway is a control-plane +# component; when it shares nodes with the workloads it serves, the default +# priority gives it no advantage under node pressure. Empty = omit the field. +priorityClassName: "" + +# -- Topology spread constraints for the gateway pod, used to spread replicas +# across zones or nodes. Empty = omit the field. +topologySpreadConstraints: [] + # -- Non-secret gateway application configuration. Top-level keys name TOML # tables and are rendered into the mounted gateway.toml file. Kubernetes # resource inputs remain outside this map; template expressions derive the diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index bbc223e32f..c8a4af6344 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -300,6 +300,8 @@ The most commonly changed values are: | `global.image.registry` / `global.image.tag` / `global.image.pullPolicy` | Shared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence. | | `gateway.image.registry` / `gateway.image.repository` / `gateway.image.tag` / `gateway.image.digest` | Gateway container image. Defaults to `ghcr.io/nvidia/openshell/gateway:latest`. | | `replicaCount` | Number of gateway replicas. Values above `1` require shared PostgreSQL through `server.externalDbSecret`. | +| `priorityClassName` | Existing Kubernetes PriorityClass for gateway pods. Empty by default. | +| `topologySpreadConstraints` | Kubernetes topology spread constraints for gateway replicas. Empty by default. Set selectors to match the gateway pod labels. | | `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. | | `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. | | `autoscaling.enabled` / `autoscaling.minReplicas` / `autoscaling.maxReplicas` | Render a HorizontalPodAutoscaler instead of a fixed `replicaCount`. Refer to [High Availability](/kubernetes/high-availability#scale-the-gateway). | diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index ec8c9dee6f..ad950b5340 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -530,6 +530,14 @@ kubectl -n openshell logs -c openshell-gateway --tail=200 `server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway pod, and the gateway propagates the effective value to sandbox supervisors. +Gateway pod scheduling accepts top-level `nodeSelector`, `affinity`, +`tolerations`, `priorityClassName`, and `topologySpreadConstraints` values for +both StatefulSet and Deployment workloads. The last two default to unset. +If a gateway pod stays Pending, inspect its scheduling events, confirm the +named PriorityClass exists (`kubectl get priorityclass`), and check that topology +spread selectors match the gateway pod labels and that eligible nodes have the +required topology labels. These values apply only to gateway pods. + #### Additional destination CA ConfigMap When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the