Repository navigation
Alert on sustained node DB CPU saturation - #390
Merged
Merged
Conversation
Reads CPU and connection-pool metrics for each configured database branch and sends a Better Stack heartbeat decision per target. A breach is 13 of the last 15 one-minute points above the threshold; points newer than 90s are ignored for ingestion lag. Too few valid points or transient API errors send nothing. Targets are config-only. Details: TC-1601
Runs the saturation check on a 5-minute cron and on manual dispatch with optional threshold and target inputs. Targets whose secrets are unset are skipped, so the workflow stays green until they are configured. Details: TC-1601
Use period=1h (17m is not a valid PlanetScale period). Judge unknown when valid points are below max(min_points, min_breach_points) so a 12-point window cannot report ok. Dedupe timestamps and keep the newest window_min points. Validate integer config fields. Redact and exit 1 on unhandled rejection. Details: TC-1601
Pool rule reads planetscale_primary_postgres_connection_state with state=active and fires at 15, matching the node's max connections. Details: TC-1601
- Parse heartbeat per target inside the try so one bad secret does not stop later targets - Redact target-derived secret names in crash output, not only the name prefixes - Document the 13-point unknown gate and the shipped 5432 pool rule - Set min_points to 13 to match min_breach_points Details: TC-1601
The check reads the database's branch list each run and uses the single branch flagged production. A missing or ambiguous production branch is reported as unknown and sends no heartbeat. The token needs read_branch on all branches of the database. Details: TC-1601
PlanetScale wraps the branch list as { data: [...] }. Read that field,
request per_page=100 so the production branch is not paged out, and
fail closed on a bare array. Details: TC-1601
Contributor
Author
|
APPROVE: see TC-1601 (reviewed exact head 9dc379f) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a scheduled check that alerts when the node database is sustained at high load.
What it checks
Every 5 minutes, for each configured database target:
planetscale_primary_pods_cpu_util_percentagesare strictly above 85.planetscale_primary_postgres_connection_state{state="active"}are at or above 15 (TINYCLOUD_DATABASE__MAX_CONNECTIONS). The shipped target uses direct Postgres, port 5432.max(min_points, min_breach_points)valid points, or no series, is unknown: no ping, so the missed-heartbeat alert fires.Only metric values are logged. Query text is never read or logged.
Heartbeat results
GETping to the Better Stack heartbeat.POSTto/failwith the target label and counts.POSTto/failwith a fixed line.Notification routing: email, the same as the Write Probe heartbeat. The Better Stack free plan has no escalation policies.
Targets and secrets
Targets live in
test/probe/db-saturation-targets.json. Adding a database is config only: a new entry plus two secrets.Secrets (names only):
PLANETSCALE_METRICS_TOKEN_NODE_DBBETTERSTACK_HEARTBEAT_URL_NODE_DBPLANETSCALE_METRICS_TOKEN_OPENKEY_DBandBETTERSTACK_HEARTBEAT_URL_OPENKEY_DBare reserved for the OpenKey DB and are not yet configured.Production branch
Each run reads the database's branch list and uses the single branch flagged as production. Metrics are read from that branch. If the list has no single production branch, or the call fails transiently, the target is unknown and sends no ping.
The token needs
read_branchon all branches of the database. The database name staystinycloud-node-production, so no target change is needed when the production branch changes.Tests
npm testintest/probe: 46 pass.npx tsc --noEmitis clean.Details: TC-1601