Skip to content

feat(deploy): add an opt-in production control-plane capacity profile - #157

Merged
rcyw merged 2 commits into
inclusionAI:developfrom
rcyw:dev/production-control-capacity
Nov 10, 2026
Merged

rcyw merged 2 commits into
inclusionAI:developfrom
rcyw:dev/production-control-capacity

Conversation

@rcyw

@rcyw rcyw commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Add an opt-in core-chart capacity profile for larger installations while preserving lightweight defaults.

The profile configures one single-active Coordinator (4/8 CPU, 16/32Gi), four combined Ingress/API replicas (4/8 CPU, 8/16Gi each), and one persistent managed Redis instance (4/8 CPU, 32/64Gi) with AOF and a 200Gi PVC. API replicas require distinct hosts and can be expanded to eight after provisioning and qualifying the required capacity. Redis gains optional node selection, tolerations and affinity so it can use a stable control pool.

values-production.yaml owns the allocations. The deployment guide explains core/umbrella/Terraform consumption, host and memory budgets, explicit SSD StorageClass selection, maintenance and rollback. Existing StatefulSet claim-template fields must be retained; expanding a bound PVC or moving storage requires a separate reviewed procedure that preserves data and credentials.

These are initial allocations, not a measured sandbox-count or connection-count guarantee. Single-instance Redis and Coordinator remain availability boundaries. No live cluster, runtime pins, credentials, API endpoints or worker scaling policies are changed.

Validation:

  • All 165 core-chart tests passed. Three focused profile/placement compatibility tests are included in test_adx_chart.py, which existing CI already executes; no separate production test runner is added.
  • The 13 Terraform ADX contract tests passed against the unchanged cloud/profile inputs.
  • Production-profile Helm lint, deployment script syntax checks, changed documentation links and diff whitespace checks passed.
  • No target-scale load test, runtime SDK regression or live deployment was performed.

rcyw added 2 commits November 9, 2026 21:17
Provide a single-active Coordinator, four API replicas with required host
separation, and persistent single-instance Redis as a sizing starting point.
Keep small-installation defaults and existing worker/runtime behavior intact.

Allow explicit Redis control-pool placement. Document SSD selection, the
four-to-eight API capacity budget, safe existing-PVC expansion and rollback,
and qualification required before claiming target sandbox or connection scale.

Validate core and umbrella rendering, default compatibility and site overrides.

Signed-off-by: wayne <rcywjo@gmail.com>
Move the profile rendering, host separation and Redis placement coverage into
the chart contract suite already executed by CI. Remove the standalone test
file and duplicated Redis/default-value assertions.

Keep the capacity guide focused on reusable deployment and storage migration,
clarify preservation of all immutable claim-template fields, and leave sizing
values in the profile without implying an unmeasured scale guarantee.

Signed-off-by: wayne <rcywjo@gmail.com>
@rcyw
rcyw requested review from WenYuLuo and hky1999 November 9, 2026 13:48

@WenYuLuo WenYuLuo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hky1999 hky1999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on 9d32699; approving.

The scope discipline is the first thing to note: this is an opt-in profile file plus three optional placement knobs on the Redis template, with lightweight defaults untouched. No live cluster, runtime pins, credentials or worker scaling change - the body says so and the diff confirms it (values-production.yaml, three with-blocks in redis.yaml, defaults of empty nodeSelector/tolerations/affinity, tests, docs).

The profile itself is sensible as a stated starting point: 1 Coordinator (4/8 CPU, 16/32Gi), 4 Ingress/API replicas (4/8, 8/16Gi) with required per-host anti-affinity matching the deployment's own selector labels, zone topologySpread as ScheduleAnyway (soft - the right choice given the hard host rule already dominates), rolling update maxSurge=0/maxUnavailable=1, and a PDB of replicas-1; Redis managed mode with AOF everysec, 200Gi PVC, 4/8 CPU, 32/64Gi, and the new placement knobs for a stable control pool. The empty storageClassName forcing an explicit SSD StorageClass override (with no portable default that could silently land on spinning disk) is the right call, and the doc says so.

The test approach - folding the three profile contract tests into the existing test_adx_chart.py rather than adding a separate production runner - is what I would have asked for. The contracts are real: the profile renders identically through core and umbrella (with node DaemonSet/ConfigMap asserted unchanged, so the profile cannot leak into worker-plane objects), resources/replicas/anti-affinity/PDB/rolling-update are asserted for both 4 and 8 replicas, and the Redis placement test renders against the baseline and asserts the placement fields merge into an otherwise byte-identical object (storage and container config preserved) - which also guards the "expanding a bound PVC or moving storage needs a separate reviewed procedure" boundary at the template level.

Verification on my side (worktree of 9d32699): the full core chart suite is 165 tests as claimed - green under root/umask-022 (CI conditions); the only two non-root failures are the pre-existing fixture issues I have flagged on PRs #122/#136/#147 (application-logs symlink test lacks the root skip guard; node-directories symlink test is umask-sensitive) - unrelated to this PR but now four times flagged, worth a dedicated cleanup commit. The 13 Terraform ADX contracts pass (they need Python 3.11+ for tomllib - environment note only). Helm lint and a template render of the profile are clean; PDBs render as expected (minAvailable 1 for Coordinator, 3 for the 4-replica API). deploy-script-check passes. CI 42628602742 on the exact head is green.

Two non-blocking observations:

  1. The honest boundary list is good (initial allocations, not measured sandbox-count guarantees; single-instance Redis and Coordinator as availability boundaries; eight API replicas requiring provisioned and qualified capacity). The doc carries these; make sure the profile file's header comment stays the first thing an operator reads, since YAML has a way of traveling without its docs.
  2. The Redis placement test asserts placement merges cleanly, but nothing yet asserts that applying the production profile to an existing release with placement already set produces a valid combination (profile + operator overrides intersect). Given values-*.yaml layering semantics, a merge test of profile-over-baseline-with-placement would close the last corner - minor.

@rcyw
rcyw merged commit 2812586 into inclusionAI:develop Nov 10, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants