Repository navigation
Add MySQL rejection and recovery metrics with bounded logs - #209
Open
pradeepkintali wants to merge 1 commit into
Open
pradeepkintali wants to merge 1 commit into
pradeepkintali wants to merge 1 commit into
Conversation
Preserve existing admission behavior and counters while exposing configured-source rejection reasons, recovery transitions, and bounded INFO logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
There was a problem hiding this comment.
🔵 Needs a closer look
Telemetry changes across admission and recovery paths introduce synchronous logging under the recovery mutex and added metric cardinality, requiring broader review of latency and operational costs.
No open issues 🎉
What changed in this PR
Adds visibility into MySQL admission rejections and recovery transitions without changing admission decisions, recovery timing, or HTTP response JSON.
Changes:
- Adds rejection counters attributed to the configured source, first blocking dependency, HTTP status, and reason.
- Adds recovery-transition counters and structured INFO logs, with rejection logs limited to once per minute per combination.
- Adds telemetry tests and documentation covering attribution, suppression, redaction, and recovery lifecycle behavior.
| File | Description |
|---|---|
pkg/throttle/throttler.go |
Initializes rejection-log throttling and reports recovery transitions. |
pkg/throttle/safety_telemetry.go |
Implements rejection and recovery counters and structured logs. |
pkg/throttle/safety_telemetry_test.go |
Tests attribution, exclusions, log suppression, redaction, and recovery transitions. |
pkg/throttle/check.go |
Records MySQL rejections and identifies specific rejection reasons. |
pkg/throttle/check_result.go |
Adds internal rejection metadata without changing JSON output. |
doc/mysql.md |
Documents metrics, log behavior, and collection limitations. |
Review effort: Balanced (adaptive)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The existing pressure gauge and recovery-active gauge show current state, but they do not explain the full admission and recovery timeline. Existing rejection counters attribute HTTP 429 responses to the blocking metric, but lack the configured source/dependency relationship and detailed HTTP 500 categories.
This change adds that visibility without changing admission decisions, thresholds, recovery timing, or HTTP response JSON.
What changes
Rejection counters: Non-internal MySQL checks returning HTTP 429 or 500 increment:
Reasons distinguish threshold violations, recovery holds, shared-domain holds, missing hosts, missing metrics, general metric errors, and other failures. Only the first blocking dependency is counted. Fallback requests use the resolved configured source name, not arbitrary caller input. Internal freno and freno-share-domain checks are excluded. Existing check.rejected counters remain unchanged.
Recovery counters: Recovery-enabled clusters emit:
Events are initialized, blocked, started, reset, and completed. Repeated readings in the same state do not create additional transition events. Initialization describes the healthy window required by a new process or leader; it does not establish overload. The existing recovery-active gauge remains unchanged. Regular replica-lag stores without a recovery duration still get rejection telemetry, but do not get recovery transitions.
Both counter families use check.* so the existing production Go Expvar mapping collects them as rates. Local counters are cumulative since process startup; Datadog exports events per second.
Logs: INFO-level key/value logs describe the same recovery transitions and admission rejections. Rejection logs are limited to one per minute per configured source/dependency/status/reason combination; counters still count every eligible rejection. Fields include the cluster, reason, value and thresholds, plus HTTP status/dependency or recovery duration as appropriate. Logs omit raw error messages, credentials and request contents, and use the existing stderr collection path for Splunk.
Validation
Using the existing golang:1.27.3 container image and vendored dependencies:
All passed. Formatting and git diff --check passed.
New tests cover rejection attribution, first-blocker counting, internal-call exclusions, configured fallback names, log suppression and redaction, recovery resets/completion, repeated-reading suppression, and leadership regain. Existing HTTP dependency/recovery and memcache tests also pass.
Scope and rollout limitations
This does not fix Datadog delivery/indexing gaps, configure Splunk, add application or individual database-host attribution, measure memcache client rejections, or implement Vitess primary discovery. The metric_error category does not establish the underlying cause. Recovery completion clears that local gate; another dependency may still reject admission.
The additional series use bounded configured-cluster dimensions, but increase telemetry cardinality. Log throttling is process-local and resets on restart. No production deployment or ingestion validation has been performed; verify both Datadog receipt and Splunk ingestion in staging before rollout.