From 69ec36a68fbe296cab1c00066ca09720ba9dff1a Mon Sep 17 00:00:00 2001 From: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com> Date: Thu, 22 Oct 2026 14:16:41 +0300 Subject: [PATCH 1/2] feat(helm): expose priorityClassName and topologySpreadConstraints The gateway pod template exposes nodeSelector, affinity and tolerations but not priorityClassName or topologySpreadConstraints. Operators who need either have to fork the chart or mutate the rendered output out-of-band. Add two optional passthroughs: - priorityClassName: the gateway is a control-plane component. When it shares nodes with the workloads it serves, the default priority gives it no advantage under node pressure, so a local capacity crunch can evict it and widen into a fleet-wide outage. - topologySpreadConstraints: spread replicas across zones or nodes. Only affinity was available, which is a blunter tool for even spreading. Both go in the shared gatewayPodTemplate, so they apply identically to the statefulset and deployment workload shapes. Both render only when set; a default render is byte-identical to before this change. The data volume size/StorageClass passthroughs from the original revision are dropped: upstream now provides them as server.dataStorageSize and server.dataStorageClass. Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com> --- deploy/helm/openshell/README.md | 2 + .../openshell/templates/_gateway-workload.tpl | 7 ++ .../tests/gateway_scheduling_test.yaml | 81 +++++++++++++++++++ deploy/helm/openshell/values.yaml | 9 +++ 4 files changed, 99 insertions(+) create mode 100644 deploy/helm/openshell/tests/gateway_scheduling_test.yaml diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index fcb7a108e4..29d1803c01 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -461,6 +461,7 @@ discovery endpoint or its TLS CA. | podLabels | object | `{}` | Extra labels to add to the gateway pod. | | podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. | | podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. | +| priorityClassName | string | `""` | PriorityClass for the gateway pod. The gateway is a control-plane component; when it shares nodes with the workloads it serves, the default priority gives it no advantage under node pressure. Empty = omit the field. | | probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. | | probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. | | probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. | @@ -592,6 +593,7 @@ discovery endpoint or its TLS CA. | supervisor.podSecurityContext.seccompProfile.type | string | `"RuntimeDefault"` | Seccomp profile applied to the shared supervisor pod. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | | tolerations | list | `[]` | Tolerations for the gateway pod. | +| topologySpreadConstraints | list | `[]` | Topology spread constraints for the gateway pod, used to spread replicas across zones or nodes. Empty = omit the field. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | | upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. | diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 12120a89ce..bb417b31a9 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -307,6 +307,9 @@ spec: {{- with .Values.server.extraVolumes }} {{- toYaml . | nindent 4 }} {{- end }} + {{- with .Values.priorityClassName }} + priorityClassName: {{ . }} + {{- end }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 4 }} @@ -319,4 +322,8 @@ spec: tolerations: {{- toYaml . | nindent 4 }} {{- end }} + {{- with .Values.topologySpreadConstraints }} + topologySpreadConstraints: + {{- toYaml . | nindent 4 }} + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/tests/gateway_scheduling_test.yaml b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml new file mode 100644 index 0000000000..f67b35437f --- /dev/null +++ b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway scheduling fields +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml + - templates/deployment.yaml +release: + name: openshell + namespace: my-namespace + +tests: + # priorityClassName — shared pod template, so both workload shapes + - it: omits priorityClassName by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.priorityClassName + + - it: sets priorityClassName on the StatefulSet when configured + template: templates/statefulset.yaml + set: + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + + - it: sets priorityClassName on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + priorityClassName: system-cluster-critical + asserts: + - equal: + path: spec.template.spec.priorityClassName + value: system-cluster-critical + + # topologySpreadConstraints — shared pod template, so both workload shapes + - it: omits topologySpreadConstraints by default on the StatefulSet + template: templates/statefulset.yaml + asserts: + - notExists: + path: spec.template.spec.topologySpreadConstraints + + - it: sets topologySpreadConstraints on the StatefulSet when configured + template: templates/statefulset.yaml + set: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].maxSkew + value: 1 + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: topology.kubernetes.io/zone + - equal: + path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable + value: DoNotSchedule + + - it: sets topologySpreadConstraints on the Deployment when configured + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + asserts: + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: kubernetes.io/hostname diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 04fc1152a4..1ca651261b 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -322,6 +322,15 @@ tolerations: [] # -- Affinity rules for the gateway pod. affinity: {} +# -- PriorityClass for the gateway pod. The gateway is a control-plane +# component; when it shares nodes with the workloads it serves, the default +# priority gives it no advantage under node pressure. Empty = omit the field. +priorityClassName: "" + +# -- Topology spread constraints for the gateway pod, used to spread replicas +# across zones or nodes. Empty = omit the field. +topologySpreadConstraints: [] + # -- Non-secret gateway application configuration. Top-level keys name TOML # tables and are rendered into the mounted gateway.toml file. Kubernetes # resource inputs remain outside this map; template expressions derive the From ac880b2a1e9d72b3ab7c9b94655e40a3426c6b97 Mon Sep 17 00:00:00 2001 From: Taylor Mutch Date: Fri, 6 Nov 2026 10:33:04 -0800 Subject: [PATCH 2/2] test(helm): cover scheduling defaults and document gateway placement Signed-off-by: Taylor Mutch --- .../tests/gateway_scheduling_test.yaml | 26 +++++++++++++++++++ docs/kubernetes/setup.mdx | 2 ++ skills/debug-openshell-cluster/SKILL.md | 8 ++++++ 3 files changed, 36 insertions(+) diff --git a/deploy/helm/openshell/tests/gateway_scheduling_test.yaml b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml index f67b35437f..3a1a755d02 100644 --- a/deploy/helm/openshell/tests/gateway_scheduling_test.yaml +++ b/deploy/helm/openshell/tests/gateway_scheduling_test.yaml @@ -11,6 +11,17 @@ release: namespace: my-namespace tests: + - it: omits scheduling overrides by default on the Deployment + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: openshell-db + asserts: + - notExists: + path: spec.template.spec.priorityClassName + - notExists: + path: spec.template.spec.topologySpreadConstraints + # priorityClassName — shared pod template, so both workload shapes - it: omits priorityClassName by default on the StatefulSet template: templates/statefulset.yaml @@ -26,6 +37,11 @@ tests: - equal: path: spec.template.spec.priorityClassName value: system-cluster-critical + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} - it: sets priorityClassName on the Deployment when configured template: templates/deployment.yaml @@ -55,6 +71,7 @@ tests: labelSelector: matchLabels: app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell asserts: - equal: path: spec.template.spec.topologySpreadConstraints[0].maxSkew @@ -65,6 +82,11 @@ tests: - equal: path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable value: DoNotSchedule + - equal: + path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels + value: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell - it: sets topologySpreadConstraints on the Deployment when configured template: templates/deployment.yaml @@ -75,6 +97,10 @@ tests: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell asserts: - equal: path: spec.template.spec.topologySpreadConstraints[0].topologyKey diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index bbc223e32f..c8a4af6344 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -300,6 +300,8 @@ The most commonly changed values are: | `global.image.registry` / `global.image.tag` / `global.image.pullPolicy` | Shared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence. | | `gateway.image.registry` / `gateway.image.repository` / `gateway.image.tag` / `gateway.image.digest` | Gateway container image. Defaults to `ghcr.io/nvidia/openshell/gateway:latest`. | | `replicaCount` | Number of gateway replicas. Values above `1` require shared PostgreSQL through `server.externalDbSecret`. | +| `priorityClassName` | Existing Kubernetes PriorityClass for gateway pods. Empty by default. | +| `topologySpreadConstraints` | Kubernetes topology spread constraints for gateway replicas. Empty by default. Set selectors to match the gateway pod labels. | | `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. | | `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. | | `autoscaling.enabled` / `autoscaling.minReplicas` / `autoscaling.maxReplicas` | Render a HorizontalPodAutoscaler instead of a fixed `replicaCount`. Refer to [High Availability](/kubernetes/high-availability#scale-the-gateway). | diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index ec8c9dee6f..ad950b5340 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -530,6 +530,14 @@ kubectl -n openshell logs -c openshell-gateway --tail=200 `server.telemetryEnabled` renders `OPENSHELL_TELEMETRY_ENABLED` on the gateway pod, and the gateway propagates the effective value to sandbox supervisors. +Gateway pod scheduling accepts top-level `nodeSelector`, `affinity`, +`tolerations`, `priorityClassName`, and `topologySpreadConstraints` values for +both StatefulSet and Deployment workloads. The last two default to unset. +If a gateway pod stays Pending, inspect its scheduling events, confirm the +named PriorityClass exists (`kubectl get priorityclass`), and check that topology +spread selectors match the gateway pod labels and that eligible nodes have the +required topology labels. These values apply only to gateway pods. + #### Additional destination CA ConfigMap When `supervisor.network.additionalCaConfigMapName` is set, Helm must mount the