Feat/dn roles catchup - #184
Draft
alix-graylog wants to merge 7 commits into
Draft
alix-graylog wants to merge 7 commits into
alix-graylog wants to merge 7 commits into
Conversation
Add opt-in role delegation so larger clusters can dedicate StatefulSets to specific OpenSearch roles (cluster_manager, data, ingest, remote_cluster_client, search) instead of every Data Node carrying all roles. - The top-level datanode block is the primary node group (legacy unsuffixed names). datanode.roles (default []) sets its OpenSearch roles and is inherited as the default by every extra group; empty keeps the Data Node default and its automatic search-role-on-snapshot behavior. - datanode.extraNodeGroups is a map keyed by group name; each value overrides the inherited datanode.* values. Keys may be hyphenated. Cross-values-file deep merge and unique names come for free. - graylog.datanode.groups enriches every group with derived fields (fullname, configmapName, pdbName, groupLabel, data/nativeLibs storage class), so the StatefulSet, ConfigMap and PDB templates iterate with `.`/`$` and never pass (root, group) dicts. The StatefulSet, ConfigMap and PDB become per-group; the headless Service and Secret stay shared, and discovery seed hosts span all groups. - Group labels (graylog-datanode-group) are emitted only when extra groups exist, so a plain install keeps today's unlabeled selector and upgrades in place. When groups are added, the primary is labeled `default` and every StatefulSet gets a distinct selector (no controller overlap). NOTE: this relabels the primary selector, which is immutable — first-time adoption requires recreating the graylog-datanode StatefulSet (documented in values.yaml). - graylog.datanode.replicas sums replicas across groups; drop the `| default N` fallbacks from graylog.replicas / graylog.datanode.replicas so values.yaml is authoritative and an explicit replicas: 0 is no longer coerced. Verified: back-compat renders identically to main (modulo random secrets and config checksum); map mode renders primary + extras with distinct selectors, per-group roles/heap/storage, group-spanning seed hosts, and falsy overrides (persistence.data.enabled: false). 66 unit tests pass; helm lint clean. Guardrails (fail without a cluster_manager group, warn on missing data role or a search group without a repo) follow in a separate commit. Co-Authored-By: Claude Opus 4.8 (1M context)[EMAIL_ADDRESS_REDACTED]om>
Add validation on top of the extraNodeGroups role delegation: - Hard fail (graylog.datanode.validate, included from the datanode StatefulSet) when no node group is eligible to be a cluster_manager, i.e. every group sets explicit roles and none includes cluster_manager. Groups with empty roles use the Data Node default (which includes cluster_manager), so the all-defaults install never trips it. - NOTES.txt warning when no group is eligible to hold data (every group sets explicit roles and none includes data). - NOTES.txt warning when a group declares the search role but no snapshot repository is configured (datanode.config.s3ClientDefault*), listing the affected groups; suppressed once a repo is set. Warnings follow the chart's existing NOTES.txt convention (fsGroup, MongoDB worker count). 7 unit tests cover the fail path, both warnings, and their negative cases via failedTemplate / matchRegexRaw; 73 tests pass, lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
Add docs/datanode-node-roles.md covering the role list, the primary group + extraNodeGroups map, a hot+search example, the guardrails, the search-tier snapshot-repository requirement, and the flat->groups migration. The migration recreates the primary StatefulSet by deleting it and upgrading in one chained step, so helm creates (not patches) the immutable selector and never errors; PVCs are retained and the primary pods restart once. Documents the role-narrowing / data-path and dedicated-manager rebuild caveats. Link the guide from the README Usage section and the DataNode values reference, and trim the values.yaml comments to a short pointer. Migration verified live on EKS: adding a data-capable group to a running flat install migrates cleanly (PVCs retained, cluster stays green as one cluster). Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>
alix-graylog
added
the
needs-discussion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a testing branch which aims to catch up the contribution from the PR #114 to current main.
More details to follow