sync to refactored break - #2
Open
brownzebra wants to merge 2467 commits into
Open
Conversation
…-workflow-logic-in-cluster-forge-to-reduce-confusion EAI-1500 update Github Workflows documentation
For medium/large clusters ArgoCD reads cluster-values/values.yaml, which the gitea-init-job rebuilds from a template rather than copying the seeded complete_values.yaml wholesale. Add an aimHardwareFamily value and emit the apps.aim-cluster-model-source.valuesObject.hardwareFamilies block into cluster-values when set, mirroring the existing airmImageRepository handling. Without it the chart fell back to the legacy install-all branch on medium/large. Part of EAI-6030.
The upstream Helm chart uses replicas: ~ (YAML null) as the default for all four Kyverno controllers. The chart's null-replica guard in _deployment.tpl only checks kindIs "invalid", which null passes — causing Kubernetes to coerce the empty rendered value to 0 replicas. This silently breaks the admission controller webhook, meaning generate policies (e.g. dynamic-pvc-creation) never fire, leaving workspace pods stuck in Pending due to missing PVCs. Set replicas: 1 explicitly for admissionController, backgroundController, cleanupController, and reportsController so all cluster-forge installs get a working Kyverno from day one. Clusters needing HA can override via their cluster-values. Upstream bug: kyverno/kyverno#8941, #6182 Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
Parameterizes the amd-gpu-operator-config DeviceConfig out-of-tree ROCm driver version by GPU family (radeon | instinct) via a gpuStackFamily / driverVersion value and a gpuStack.driverVersion helper. Empty resolves to instinct (7.0), so existing installs are unchanged; radeon resolves to the ROCm 7.13 tech-preview pin. The GPU Operator chart version is selected separately through the app-level apps.amd-gpu-operator.path field. cluster-bloom injects both the path and the config valuesObject: small clusters via --set on the helm render, medium/large via the gitea-init-job, which now emits the amd-gpu-operator path and amd-gpu-operator-config valuesObject into cluster-values. Radeon driver version is a placeholder pending EAI-5906. Part of EAI-6030.
Split the previously unversioned amd-gpu-operator-config chart into v1.4.1 (the existing DeviceConfig example, default) and v1.5.1-beta.0 (the new-schema DeviceConfig for the radeon tech-preview stack). The beta DeviceConfig resolves its ROCm driver version through the existing gpuStack.driverVersion helper, so radeon gets 7.13 and the default stays 7.0. root/values.yaml defaults the config app path to amd-gpu-operator-config/v1.4.1; the beta chart is selected only when cluster-bloom injects apps.amd-gpu-operator-config.path for GPU_STACK_FAMILY=radeon. The gitea-init-job now emits that path for medium/large clusters. Part of EAI-6030.
Read the imageRegistrySecret name from values.yaml instead of hardcoding dockerhub-amdpsdo-regcred across the four DeviceConfig component sections. Default keeps the existing secret name. Part of EAI-6030.
Guard each imageRegistrySecret block with the resolved name so an empty or absent imageRegistrySecret.name renders no key at all, instead of emitting an empty name that produces invalid DeviceConfig YAML. Part of EAI-6030.
* feat(extra-apps): Add blueprint for vllm-sr Example blueprint for adding vllm-sr as an extra app. The config for it is just a placeholder for demonstrating the idea. The user is required to just use these as reference and implement these in the per cluster cluster-values repo. * redo docs * fix(dashboard): update dashboard docs and router access * feat: retries and timeouts * more retry docs
feat(EAI-8292): webhook heal script and platform gates (complements #824)
Explain the Gitea and Argo CD workflow for adding, validating, replacing, and removing additional model sources, and index repository documentation. Co-authored-by: Cursor <cursoragent@cursor.com>
Align filename with underscore convention used by other docs/. Co-authored-by: Cursor <cursoragent@cursor.com>
Expand aim_model_management.md with packaged baseline, base catalog, and version policy. Add adding_aim_catalog_models.md as a streamlined admin procedure. Update README links. Co-authored-by: Cursor <cursoragent@cursor.com>
Clarify aim_model_management.md is customer-facing and not tied to release cadence; rename inter-release wording to cluster-managed additions. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace private repo references in customer-facing docs with the public amd-enterprise-ai/aim-build repository and remove private-repo path details. Co-authored-by: Cursor <cursoragent@cursor.com>
…rdwareFamilies helm value
…urce docs: document inter-release AIM catalog management
Cluster-auth has been removed from the stack. Replaced with envoy SecurityPolicy in the blueprint.
sanity check to get script if missing from release
fix(blueprint): vllm-sr remove cluster-auth references
Match hardwareFamilies empty vs profiles behaviour to legacy.yaml and profiles.yaml on main; drop the claim that empty list is unsupported. Co-authored-by: Cursor <cursoragent@cursor.com>
…yaml Empty hardwareFamilies is an unfiltered catalog (Instinct 0.11.1+ plus mixed bases), not the old generic release set. Update chart comments and operator docs to match. Co-authored-by: Cursor <cursoragent@cursor.com>
…urce docs: align AIM catalog lifecycle with in-tree chart templates
The body-aware extAuth on ai-gateway buffers the request body at maxRequestBytes=64KiB before authorizing header-less (model-in-body) requests. Envoy returns 413 for any larger body (skipping auth, and overriding failOpen), so standard OpenAI/Anthropic clients sending 200KB-2MB contexts are rejected gateway-wide, for every model. Parameterize as aiGateway.bodyAuthMaxRequestBytes (default 32MiB) so large-context and image-bearing inference turns are not rejected; the value sits under the 50Mi connection bufferLimit and is tunable per env. EAI-8489
Follow-up on the 32MiB commit on this branch. - 4MiB, not 32MiB. The ai-gateway-discovery authz pod that receives the buffered body is capped at limits.memory 256Mi, and an in-flight header-less request is held both by Envoy and by that handler, so 32MiB is more than the current deployment can absorb. 4MiB covers a ~200k-token turn (~1MB) with headroom. - Drive both charts from global.aiGateway.bodyAuthMaxRequestBytes. The gateway-scoped policy here is not the only cap on the header-less path: ai-gateway-discovery stamps per-model catch-all SecurityPolicies (sectionName route-not-found) that are rule-scoped and override this one per route. They carry their own ceiling, so the two must move together. - Render through int64 and fail on a non-positive value. A bare number in a values file is a float64, so the root wiring emitted 4.194304e+06 and int64 casts that to 0, which would have published maxRequestBytes: 0 instead of failing. The matching ai-gateway-discovery change lands in silogen/core.
Raise ai-gateway body-authz request-body cap to 32MiB (EAI-8489)
The AIM catalog docs described behaviour that only exists in unmerged work, so operators reading them would configure the wrong thing. - Drop AIM_HARDWARE_FAMILY auto-detection claims from the lifecycle doc and chart README; auto-detection is not in cluster-bloom main, where the setting has no default and is injected only when set. Document that an install leaving it unset lands on the unfiltered catalog. - Replace the "published into the chart" source-of-truth framing with the actual manual copy at release time, and drop the dead public aim-build link. Point operators at cluster-managed additions when they need an AIM version before the next release. - Note that aim-cluster-model-source-additional is not shipped and needs one-time operator setup. - Fix stale "legacy" naming in root/values.yaml after the legacy.yaml -> unfiltered.yaml rename. Co-authored-by: Cursor <cursoragent@cursor.com>
AIM Engine discovery is append-only, so dropping an image from a source filter leaves discovered models in place, while deleting the source CR garbage-collects them and can break running deployments. Empty filters are invalid (MinItems=1). Packaged source names are a stable API. Co-authored-by: Cursor <cursoragent@cursor.com>
…ities Operators see the packaged baseline as the Cluster Forge release they installed. Who authored the internal lists is not needed to operate a cluster. Co-authored-by: Cursor <cursoragent@cursor.com>
…talog_state docs(aim-catalog): describe shipped catalog behaviour, not pending work
The root Application is rendered two times. bloom renders it one time
with helm --set, and ArgoCD then renders the same chart again to manage
the app-of-apps. The second render reads only the value files, so each
value that was given with --set and is not repeated as a parameter is
lost, and the root app overwrites its own spec without it.
global.domain and global.clusterSize were lost this way. An empty
domain made envoy-gateway-config write the CoreDNS rewrite
`regex .*\.`, which matches every name, so each name in the cluster
resolved to the gateway service. The ArgoCD controller could not reach
its repo-server, all 39 applications stayed in ComparisonError, and no
secret was made. An empty clusterSize dropped values_<size>.yaml from
the value files, so the cluster got the wrong set of applications.
Only the small path is affected. With externalValues.enabled the two
values come from cluster-values/values.yaml in the in-cluster Gitea.
To see the difference:
helm template cluster-forge root --show-only templates/cluster-forge.yaml \
--values root/values.yaml --values root/values_small.yaml \
--set global.domain=1.2.3.4.nip.io | grep -A1 global.domain
The certgen hooks carry "helm.sh/hook" but no delete policy. ArgoCD reads a missing policy as BeforeHookCreation, so it deletes each hook before it makes the hook again, and it does that each time it resumes the operation. The certgen ServiceAccount is a hook with weight -1, so it must exist before the certgen job can make a pod. ArgoCD deletes that ServiceAccount on each resume. The Job object stays, but each attempt to make its pod fails, so the job never completes. envoy-gateway stays at PreSync with "waiting for completion of hook", the Gateway API CRDs are never installed, and each application that uses a Gateway, an HTTPRoute or a TLSRoute cannot sync. Measured on a one-node cluster with CLUSTER_SIZE small, and with the root app domain fix in place: the job stayed at 0/1 completions for 4 minutes 35 seconds, the namespace held only the default service account, no Gateway API CRD was installed, and the job controller repeated ten times: Error creating: pods "envoy-gateway-gateway-helm-certgen-" is forbidden: error looking up service account envoy-gateway-system/envoy-gateway-gateway-helm-certgen: serviceaccount "envoy-gateway-gateway-helm-certgen" not found Each cluster size uses this chart, but the failure is a race between the interval of two resumes and the time the certgen job needs. A one-node cluster with CLUSTER_SIZE medium completed the job before the next resume, with this change and also without it, so only the small path shows the failure. The change stays correct for each size. hook-succeeded keeps the hooks until the job completes, and removes them after it.
bloom does not bootstrap gitea when CLUSTER_SIZE is small. The task
clusterforge_setup.yaml holds `when: (CLUSTER_SIZE | default('medium'))
!= 'small'`, and bootstrap_gitea.yaml is the only thing that makes the
secret gitea-admin-credentials.
values_small.yaml enabled the gitea and gitea-config applications all the
same, so the pod waited for a secret that nothing makes. Measured on a
one-node cluster: `secret "gitea-admin-credentials" not found`, 255
restarts in 59 minutes, and the application stayed Degraded.
The small path reads the repository from GitHub, because
externalValues.enabled is false and clusterForge.repoUrl names GitHub.
It therefore has no use for the in-cluster Gitea, and nothing else in
values_small.yaml names it.
EAI-8229 Fix CLUSTER_SIZE small: keep global.domain, hooks, drop gitea
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.