From 2f36c85722adace0988fe4487da2408d7d8ec964 Mon Sep 17 00:00:00 2001 From: Ramon Marquez Date: Sun, 12 Jul 2026 19:19:54 +0200 Subject: [PATCH 1/2] fix(datanode): raise default data volume so search cache fits when tiering enabled The default datanode.config.nodeSearchCacheSize (10gb) exceeded the default datanode data volume (8Gi, ~7gb usable). node_search_cache_size reserves disk on the datanode data volume, so once an S3/path_repo snapshot repository is configured the datanode fails preflight ("not enough usable space for the node search cache") and CrashLoops. Plain installs are unaffected because the check only runs when a repository is enabled. - values.yaml: datanode.persistence.data.size 8Gi -> 20Gi so the default cache fits alongside hot index data. - NOTES.txt: warn at install/upgrade time when tiering is enabled and the cache does not fit the data volume (skipped when an existingClaim is used). Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]> --- charts/graylog/templates/NOTES.txt | 9 +++++++++ charts/graylog/values.yaml | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/charts/graylog/templates/NOTES.txt b/charts/graylog/templates/NOTES.txt index 947b623..1143e8f 100644 --- a/charts/graylog/templates/NOTES.txt +++ b/charts/graylog/templates/NOTES.txt @@ -26,6 +26,15 @@ SUMMARY volume ownership (e.g. AWS EBS). See the "Hardened Environments" section of the README before opting out. {{- end }} + {{- $cache := .Values.datanode.config.nodeSearchCacheSize | regexFind "^[0-9]+" | int }} + {{- $disk := .Values.datanode.persistence.data.size | regexFind "^[0-9]+" | int }} + {{- if and .Values.datanode.config.s3ClientDefaultEndpoint (not .Values.datanode.persistence.data.existingClaim) (ge $cache $disk) }} + + WARNING: nodeSearchCacheSize ({{ .Values.datanode.config.nodeSearchCacheSize }}) does not fit in the Datanode + data volume ({{ .Values.datanode.persistence.data.size }}). Increase datanode.persistence.data.size, or the + Datanode will fail preflight when data tiering is enabled. + {{- end }} + Use the following command to list all the resources deployed with this release: helm get all {{ .Release.Name }} -n {{ .Release.Namespace }} diff --git a/charts/graylog/values.yaml b/charts/graylog/values.yaml index 9bb445f..50bd329 100644 --- a/charts/graylog/values.yaml +++ b/charts/graylog/values.yaml @@ -253,7 +253,7 @@ datanode: existingClaim: "" mountPath: "" accessModes: [] - size: "8Gi" + size: "20Gi" annotations: {} labels: {} selector: {} From 5df35de2435ae24cea49ad6aa1222b51941bd700 Mon Sep 17 00:00:00 2001 From: Ramon Marquez Date: Sun, 12 Jul 2026 20:07:03 +0200 Subject: [PATCH 2/2] docs(datanode): document data tiering setup (S3 + MinIO) - README: add an "Enable Data Tiering" usage section covering the S3 client config, the AWS https/path-style overrides, and the cache-vs-data-volume sizing note; add ToC entries for Geolocation and Data Tiering; fix a broken TLS anchor; use absolute URLs for repo doc links so they resolve on ArtifactHub. - docs/data-tiering.md: new guide with AWS S3 and MinIO walkthroughs, handing off to go2docs for the Enterprise warm-tier UI workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]> --- charts/graylog/README.md | 42 ++++++++++++- docs/data-tiering.md | 124 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+), 3 deletions(-) create mode 100644 docs/data-tiering.md diff --git a/charts/graylog/README.md b/charts/graylog/README.md index dc4cdb1..7619664 100644 --- a/charts/graylog/README.md +++ b/charts/graylog/README.md @@ -23,6 +23,8 @@ Official Helm chart for Graylog. * [Customize deployed Kubernetes resources](#customize-deployed-kubernetes-resources) * [Add inputs](#add-inputs) * [Enable TLS](#enable-tls) + * [Enable Geolocation](#enable-geolocation) + * [Enable Data Tiering](#enable-data-tiering) * [Using External Resources](#using-external-resources) * [Managing Secrets Externally](#managing-secrets-externally) * [Bring Your Own MongoDB](#bring-your-own-mongodb) @@ -70,7 +72,7 @@ You can use any ingress controller (e.g., NGINX, HAProxy), but make sure it's in ### cert-manager -You can always [bring your own certificates](#bring-your-own-certificate-ingress-controller-recommended), +You can always [bring your own certificates](#option-1-bring-your-own-certificate-with-ingress-controller-recommended), but using `cert-manager` can simplify TLS setup and certificate renewal considerably. Make sure you have [Ingress Controller](#ingress-controller) installed, and that `ingress.enabled` is set to `true`. @@ -388,6 +390,40 @@ Use the following paths when enabling the Geo-location processor in the Graylog - Path to the city database: `/usr/share/graylog/data/geolocation/GeoLite2-City.mmdb` - Path to the ASN database: `/usr/share/graylog/data/geolocation/GeoLite2-ASN.mmdb` +## Enable Data Tiering + +Data tiering moves older index data into a lower-cost **warm tier** backed by S3-compatible object storage, where it is kept as +searchable snapshots. This is a [Graylog Enterprise](https://go2docs.graylog.org/current/setting_up_graylog/data_tiering.htm) +feature and requires an Enterprise license. + +The chart configures the DataNode S3 client through the `datanode.config.s3ClientDefault*` values: `s3ClientDefaultEndpoint`, +`s3ClientDefaultAccessKey`, and `s3ClientDefaultSecretKey` must all be set together. + +The defaults `s3ClientDefaultProtocol: "http"`, `s3ClientDefaultPathStyleAccess: "true"` target S3-compatible stores such as MinIO. +For actual AWS S3, override the protocol and addressing style: + +```sh +helm upgrade -i graylog graylog/graylog -n graylog --reuse-values \ + --set datanode.config.s3ClientDefaultEndpoint="s3..amazonaws.com" \ + --set datanode.config.s3ClientDefaultRegion="" \ + --set datanode.config.s3ClientDefaultProtocol="https" \ + --set datanode.config.s3ClientDefaultPathStyleAccess="false" \ + --set datanode.config.s3ClientDefaultAccessKey="" \ + --set datanode.config.s3ClientDefaultSecretKey="" +``` + +> [!IMPORTANT] +> `datanode.config.nodeSearchCacheSize` reserves space on the DataNode data volume for the searchable-snapshot cache. +> It must fit within `datanode.persistence.data.size` alongside hot index data, or the DataNode fails preflight with +> `not enough usable space for the node search cache`. Keep the data volume comfortably larger than the cache. + +Once the Data Node is running, create the warm-tier repository and enable the warm tier on an index set from the Graylog web UI. +See [Create a Warm Tier on Data Node](https://go2docs.graylog.org/current/setting_up_graylog/create_warm_tier_on_data_node.htm) +for the full workflow. + +For step-by-step walkthroughs of both AWS S3 and MinIO, see +the [Data Tiering guide](https://github.com/Graylog2/graylog-helm/blob/main/docs/data-tiering.md). + # Using External Resources ## Managing Secrets Externally @@ -451,8 +487,8 @@ namespace to `baseline` rather than `restricted`, or apply the exemptions above. ## Back Up and Restore MongoDB -See [the included guide](../../docs/mongodb-backup-restore.md) if you need to take a manual `mongodump` backup of -Graylog's MongoDB database and restore it with `mongorestore`. +See [the included guide](https://github.com/Graylog2/graylog-helm/blob/main/docs/mongodb-backup-restore.md) if you need +to take a manual `mongodump` backup of Graylog's MongoDB database and restore it with `mongorestore`. # Uninstall ```sh diff --git a/docs/data-tiering.md b/docs/data-tiering.md new file mode 100644 index 0000000..c11c6e6 --- /dev/null +++ b/docs/data-tiering.md @@ -0,0 +1,124 @@ +# Data Tiering (Warm Tier) + +Graylog [data tiering](https://go2docs.graylog.org/current/setting_up_graylog/data_tiering.htm) moves older +index data into a lower-cost **warm tier** backed by S3-compatible object storage, where it is kept as +*searchable snapshots*. That way data stays queryable without living on hot storage. + +This guide covers the chart-side configuration: pointing the DataNode S3 client at a bucket. Creating the +warm-tier repository and enabling the warm tier on an index set is done afterward in the Graylog web UI (see +[Create a Warm Tier on Data Node](https://go2docs.graylog.org/current/setting_up_graylog/create_warm_tier_on_data_node.htm)). + +> [!NOTE] +> Data tiering is a **Graylog Enterprise** feature and requires an Enterprise license (add it in the Graylog web +> UI under *System > Licenses*). + +## How the chart fits in + +The chart configures the Data Node S3 client through the `datanode.config.s3ClientDefault*` values. These map +directly to the Data Node's `s3_client_default_*` settings and are written into the searchable-snapshot +repository configuration. + +| Value | Purpose | Default | +| --- | --- | --- | +| `s3ClientDefaultEndpoint` | S3 endpoint, host or `host:port`, no scheme | `""` | +| `s3ClientDefaultRegion` | S3 region | `"us-east-2"` | +| `s3ClientDefaultProtocol` | `http` or `https` | `"http"` | +| `s3ClientDefaultPathStyleAccess` | Path-style (`true`) vs virtual-hosted (`false`) addressing | `"true"` | +| `s3ClientDefaultAccessKey` | Access key (stored in a Secret) | `""` | +| `s3ClientDefaultSecretKey` | Secret key (stored in a Secret) | `""` | +| `nodeSearchCacheSize` | On-disk cache for warm-tier data on the Data Node data volume | `"10gb"` | + +> [!IMPORTANT] +> `s3ClientDefaultEndpoint`, `s3ClientDefaultAccessKey`, and `s3ClientDefaultSecretKey` must all be set together. +> Setting only some of them fails template rendering by design. + +> [!IMPORTANT] +> `nodeSearchCacheSize` reserves space on the Data Node **data volume** (disk, not memory), *not* RAM. It must fit +> within `datanode.persistence.data.size` alongside hot index data, or the Data Node fails preflight with +> `not enough usable space for the node search cache` and crash-loops. Keep the data volume comfortably larger +> than the cache (the chart default pairs a `10gb` cache with a `20Gi` data volume). + +> [!NOTE] +> The default `s3ClientDefaultProtocol: "http"` + `s3ClientDefaultPathStyleAccess: "true"` suit S3-compatible +> stores such as MinIO. **Real AWS S3 requires overrides** (see below). + +## Option A — Amazon S3 + +Create a bucket and an IAM user with read/write access to it, then configure the chart. For AWS you must set +`s3ClientDefaultProtocol: "https"` and `s3ClientDefaultPathStyleAccess: "false"` (AWS uses virtual-hosted-style +addressing). + +```yaml +datanode: + config: + s3ClientDefaultEndpoint: "s3.us-east-1.amazonaws.com" # host only, no scheme + s3ClientDefaultRegion: "us-east-1" # must match the bucket region + s3ClientDefaultProtocol: "https" + s3ClientDefaultPathStyleAccess: "false" + s3ClientDefaultAccessKey: "" + s3ClientDefaultSecretKey: "" +``` + +A minimal least-privilege IAM policy for the bucket: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads", "s3:ListBucketVersions"], + "Resource": "arn:aws:s3:::" + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"], + "Resource": "arn:aws:s3:::/*" + } + ] +} +``` + +## Option B — MinIO (S3-compatible) + +Deploy MinIO in the cluster (e.g. via the [MinIO Helm chart](https://github.com/minio/minio/tree/master/helm/minio)) +and create a bucket. Keep `http` and path-style access, and point the endpoint at the MinIO service: + +```yaml +datanode: + config: + s3ClientDefaultEndpoint: "minio:9000" # .:, no scheme + s3ClientDefaultRegion: "us-east-1" + s3ClientDefaultProtocol: "http" # default + s3ClientDefaultPathStyleAccess: "true" # default + s3ClientDefaultAccessKey: "" + s3ClientDefaultSecretKey: "" +``` + +## Enable the warm tier + +After the Data Node restarts with the S3 configuration, finish the setup in the Graylog web UI: + +1. Go to *System > Indices* and edit an index set. +2. Under *Rotation and Retention*, select *Data Tiering* and create a warm storage repository (type **Amazon S3**), + giving it a name, your bucket, and a base path. +3. Enable the warm tier and choose the repository. + +Full instructions: [Create a Warm Tier on Data Node](https://go2docs.graylog.org/current/setting_up_graylog/create_warm_tier_on_data_node.htm). + +> [!TIP] +> When entering the repository **base path**, avoid a trailing slash (use `snapshots`, not `snapshots/`). A trailing +> slash produces double-slash object keys; some S3-compatible stores (e.g. MinIO) reject this with `Object name contains +> unsupported characters`. + +## Verify + +- Confirm the Data Node picked up the S3 configuration: + + ```sh + kubectl -n graylog logs | grep "S3 repository configured" + ``` + +- In the web UI, the warm storage repository should show as ready after creation. Once an index rolls over to the + warm tier, snapshot objects appear under the base path in your bucket, and the index shows a `warm` badge while + remaining searchable. \ No newline at end of file