Repository navigation
feat(deploy): add an opt-in production control-plane capacity profile - #157
Conversation
Provide a single-active Coordinator, four API replicas with required host separation, and persistent single-instance Redis as a sizing starting point. Keep small-installation defaults and existing worker/runtime behavior intact. Allow explicit Redis control-pool placement. Document SSD selection, the four-to-eight API capacity budget, safe existing-PVC expansion and rollback, and qualification required before claiming target sandbox or connection scale. Validate core and umbrella rendering, default compatibility and site overrides. Signed-off-by: wayne <rcywjo@gmail.com>
Move the profile rendering, host separation and Redis placement coverage into the chart contract suite already executed by CI. Remove the standalone test file and duplicated Redis/default-value assertions. Keep the capacity guide focused on reusable deployment and storage migration, clarify preservation of all immutable claim-template fields, and leave sizing values in the profile without implying an unmeasured scale guarantee. Signed-off-by: wayne <rcywjo@gmail.com>
hky1999
left a comment
There was a problem hiding this comment.
Verified on 9d32699; approving.
The scope discipline is the first thing to note: this is an opt-in profile file plus three optional placement knobs on the Redis template, with lightweight defaults untouched. No live cluster, runtime pins, credentials or worker scaling change - the body says so and the diff confirms it (values-production.yaml, three with-blocks in redis.yaml, defaults of empty nodeSelector/tolerations/affinity, tests, docs).
The profile itself is sensible as a stated starting point: 1 Coordinator (4/8 CPU, 16/32Gi), 4 Ingress/API replicas (4/8, 8/16Gi) with required per-host anti-affinity matching the deployment's own selector labels, zone topologySpread as ScheduleAnyway (soft - the right choice given the hard host rule already dominates), rolling update maxSurge=0/maxUnavailable=1, and a PDB of replicas-1; Redis managed mode with AOF everysec, 200Gi PVC, 4/8 CPU, 32/64Gi, and the new placement knobs for a stable control pool. The empty storageClassName forcing an explicit SSD StorageClass override (with no portable default that could silently land on spinning disk) is the right call, and the doc says so.
The test approach - folding the three profile contract tests into the existing test_adx_chart.py rather than adding a separate production runner - is what I would have asked for. The contracts are real: the profile renders identically through core and umbrella (with node DaemonSet/ConfigMap asserted unchanged, so the profile cannot leak into worker-plane objects), resources/replicas/anti-affinity/PDB/rolling-update are asserted for both 4 and 8 replicas, and the Redis placement test renders against the baseline and asserts the placement fields merge into an otherwise byte-identical object (storage and container config preserved) - which also guards the "expanding a bound PVC or moving storage needs a separate reviewed procedure" boundary at the template level.
Verification on my side (worktree of 9d32699): the full core chart suite is 165 tests as claimed - green under root/umask-022 (CI conditions); the only two non-root failures are the pre-existing fixture issues I have flagged on PRs #122/#136/#147 (application-logs symlink test lacks the root skip guard; node-directories symlink test is umask-sensitive) - unrelated to this PR but now four times flagged, worth a dedicated cleanup commit. The 13 Terraform ADX contracts pass (they need Python 3.11+ for tomllib - environment note only). Helm lint and a template render of the profile are clean; PDBs render as expected (minAvailable 1 for Coordinator, 3 for the 4-replica API). deploy-script-check passes. CI 42628602742 on the exact head is green.
Two non-blocking observations:
- The honest boundary list is good (initial allocations, not measured sandbox-count guarantees; single-instance Redis and Coordinator as availability boundaries; eight API replicas requiring provisioned and qualified capacity). The doc carries these; make sure the profile file's header comment stays the first thing an operator reads, since YAML has a way of traveling without its docs.
- The Redis placement test asserts placement merges cleanly, but nothing yet asserts that applying the production profile to an existing release with placement already set produces a valid combination (profile + operator overrides intersect). Given values-*.yaml layering semantics, a merge test of profile-over-baseline-with-placement would close the last corner - minor.
Add an opt-in core-chart capacity profile for larger installations while preserving lightweight defaults.
The profile configures one single-active Coordinator (4/8 CPU, 16/32Gi), four combined Ingress/API replicas (4/8 CPU, 8/16Gi each), and one persistent managed Redis instance (4/8 CPU, 32/64Gi) with AOF and a 200Gi PVC. API replicas require distinct hosts and can be expanded to eight after provisioning and qualifying the required capacity. Redis gains optional node selection, tolerations and affinity so it can use a stable control pool.
values-production.yamlowns the allocations. The deployment guide explains core/umbrella/Terraform consumption, host and memory budgets, explicit SSD StorageClass selection, maintenance and rollback. Existing StatefulSet claim-template fields must be retained; expanding a bound PVC or moving storage requires a separate reviewed procedure that preserves data and credentials.These are initial allocations, not a measured sandbox-count or connection-count guarantee. Single-instance Redis and Coordinator remain availability boundaries. No live cluster, runtime pins, credentials, API endpoints or worker scaling policies are changed.
Validation:
test_adx_chart.py, which existing CI already executes; no separate production test runner is added.