feat(alerting): cluster event log alert rules for docker clusters

[boddumanohar]

Ports simplyblock-operator#474 to docker-based clusters.

Adds sbctl cluster event-alerts <cluster-id>, which provisions eight Grafana alert rules that read the cluster event log through /api/v2/clusters/<id>/logs instead of the Thanos metrics the rules in alerting/alert_rules.yaml read.

The event log carries the transition an entity made, not only the state it ended in. That is what lets these rules tell an operator's shutdown apart from a fault that ended in the same state, and lets them report three conditions no metric carries: a device removal and the two journal-compression conditions.

How it works

sudo sbctl cluster event-alerts <CLUSTER_ID>            # enable
sudo sbctl cluster event-alerts <CLUSTER_ID> --disable  # remove

Run once, from any management node. The command:

1. reads Grafana's provisioning paths off the running monitoring_grafana service, rather than computing them from this package's location;
2. renders event_alert_rules.yaml and datasource-events.yml for this cluster;
3. writes both onto every management node through that node's docker API (Grafana is constrained to node.role == manager and may be rescheduled to any of them, and these are host paths);
4. recreates the Grafana task with --force, adding the plugin env and the data source mount, so Grafana re-reads provisioning.

Opt-in because the rules need a REST data source, and therefore a Grafana plugin the deployed image does not carry and downloads once (~74 MB) into the grafana_data volume. --plugin-preinstalled covers an air-gapped install.

Deliberately not wired into cluster create or docker-compose-swarm-monitoring.yml: nothing on the create path changes, so an upgrade cannot alter the behaviour of a cluster whose operator does not run the command. It also means this works on clusters that already exist, which for docker is all of them.

The rules

Alert	Severity	Fires while	Suppressed when
StorageNode_left_online	critical	a node is in any state other than online/in_creation	it reached in_shutdown, in_removal, pending_removal or removed
Device_became_unavailable	warning	a device is unavailable	it went down within 120s of a node leaving online
Device_removed	critical	a device reached removed in the last 24h	—
Cluster_became_degraded	warning	the cluster is degraded	a node is down through an operator-requested path
Cluster_became_suspended	critical	the cluster is suspended	—
Cluster_capacity_reached	warning	newest absolute/provisioned capacity event is Warning or Critical	—
JM_records_threshold_exceeded	critical	a journal reported a compression backlog in the last hour	—
JM_compression_error	critical	newest compression event for a node+journal is a failure	—

Each rule returns one row per entity that is currently wrong, so healing is structural: the reverting event removes the row, and noDataState: OK turns an empty result into resolved.

Known gaps, for the reviewer

• Overlap. Five of the eight duplicate a rule in alert_rules.yaml, and both sets route to the same contact point with group_by: ['grafana_folder','alertname'], so those conditions notify twice. Retiring the duplicates is a separate change.
• Three of those duplicates (StorageNode_status_online_to_down, ..._to_unreachable, Device_status_online_to_unavailable) can only ever fire via noDataState today: simplyblock_web/api/v1/metrics.py:141 skips any node that is not ONLINE and :201 any device that is not online/read_only/cannot_allocate, so snode_status_code is always 0 and device_status_code never carries unavailable (3) or removed (4). Worth fixing on its own — it also makes those values wrong in every dashboard.
• JM_records_threshold_exceeded clears on a one-hour timer, not on recovery. check_jm_compression_backlog logs the downward crossing but writes no event, so there is nothing for the fold to see. ~10 lines to emit one, then the rule can heal structurally.
• A planned storage-node restart/suspend still notifies, since in_restart and suspended are not in the operator-requested set.
• Cascade suppression is time-based, not per node. A device event's log record carries no parent node (device_events.py:23 sets node_id to the device id, and get_logs exposes no owning node), so a device failure on another node within 120s of a node going down is suppressed too.
• The rules now exist twice, here and in the Helm chart, with nothing keeping them in step. A single source of truth would be the durable fix.

Testing

tox run-parallel -e lint,types and tox run -e unit are green, with one pre-existing unrelated failure (test_leaderless_warning_is_rate_limited, which reproduces on origin/main — its rate limiter uses 0.0 as a sentinel against time.monotonic()).

Not yet exercised against a live cluster — that is what this draft is for. On a real cluster, worth checking:

# every rule in group simplyblock_events should report health=ok
curl -s -u admin:<CLUSTER_SECRET> "http://<MGMT_IP>/grafana/api/prometheus/grafana/api/v1/rules"

then trigger a device removal and a node fault and confirm both the firing and the healing, and that a clean storage-node shutdown stays quiet.

🤖 Generated with Claude Code