Skip to content

Add MySQL rejection and recovery metrics with bounded logs - #209

Open
pradeepkintali wants to merge 1 commit into
masterfrom
pkintali/safety-transition-telemetry
Open

pradeepkintali wants to merge 1 commit into
masterfrom
pkintali/safety-transition-telemetry

Conversation

@pradeepkintali

Copy link
Copy Markdown
Contributor

Why

The existing pressure gauge and recovery-active gauge show current state, but they do not explain the full admission and recovery timeline. Existing rejection counters attribute HTTP 429 responses to the blocking metric, but lack the configured source/dependency relationship and detailed HTTP 500 categories.

This change adds that visibility without changing admission decisions, thresholds, recovery timing, or HTTP response JSON.

What changes

Rejection counters: Non-internal MySQL checks returning HTTP 429 or 500 increment:

check.rejection.mysql.<configured-source>.dependency.<blocking-cluster>.status.<status>.<reason>.total

Reasons distinguish threshold violations, recovery holds, shared-domain holds, missing hosts, missing metrics, general metric errors, and other failures. Only the first blocking dependency is counted. Fallback requests use the resolved configured source name, not arbitrary caller input. Internal freno and freno-share-domain checks are excluded. Existing check.rejected counters remain unchanged.

Recovery counters: Recovery-enabled clusters emit:

check.recovery.mysql.<cluster>.<event>.total

Events are initialized, blocked, started, reset, and completed. Repeated readings in the same state do not create additional transition events. Initialization describes the healthy window required by a new process or leader; it does not establish overload. The existing recovery-active gauge remains unchanged. Regular replica-lag stores without a recovery duration still get rejection telemetry, but do not get recovery transitions.

Both counter families use check.* so the existing production Go Expvar mapping collects them as rates. Local counters are cumulative since process startup; Datadog exports events per second.

Logs: INFO-level key/value logs describe the same recovery transitions and admission rejections. Rejection logs are limited to one per minute per configured source/dependency/status/reason combination; counters still count every eligible rejection. Fields include the cluster, reason, value and thresholds, plus HTTP status/dependency or recovery duration as appropriate. Logs omit raw error messages, credentials and request contents, and use the existing stderr collection path for Splunk.

Validation

Using the existing golang:1.27.3 container image and vendored dependencies:

go test -mod=vendor -race ./pkg/... ./cmd/...
go vet -mod=vendor ./pkg/throttle
go build -mod=vendor -o /tmp/freno ./cmd/freno

All passed. Formatting and git diff --check passed.

New tests cover rejection attribution, first-blocker counting, internal-call exclusions, configured fallback names, log suppression and redaction, recovery resets/completion, repeated-reading suppression, and leadership regain. Existing HTTP dependency/recovery and memcache tests also pass.

Scope and rollout limitations

This does not fix Datadog delivery/indexing gaps, configure Splunk, add application or individual database-host attribution, measure memcache client rejections, or implement Vitess primary discovery. The metric_error category does not establish the underlying cause. Recovery completion clears that local gate; another dependency may still reject admission.

The additional series use bounded configured-cluster dimensions, but increase telemetry cardinality. Log throttling is process-local and resets on restart. No production deployment or ingestion validation has been performed; verify both Datadog receipt and Splunk ingestion in staging before rollout.

Preserve existing admission behavior and counters while exposing configured-source rejection reasons, recovery transitions, and bounded INFO logs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Copilot AI adaptive review requested due to automatic review settings November 21, 2026 06:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Telemetry changes across admission and recovery paths introduce synchronous logging under the recovery mutex and added metric cardinality, requiring broader review of latency and operational costs.

No open issues 🎉

What changed in this PR

Adds visibility into MySQL admission rejections and recovery transitions without changing admission decisions, recovery timing, or HTTP response JSON.

Changes:

  • Adds rejection counters attributed to the configured source, first blocking dependency, HTTP status, and reason.
  • Adds recovery-transition counters and structured INFO logs, with rejection logs limited to once per minute per combination.
  • Adds telemetry tests and documentation covering attribution, suppression, redaction, and recovery lifecycle behavior.
File Description
pkg/​throttle/​throttler.go Initializes rejection-log throttling and reports recovery transitions.
pkg/​throttle/​safety_telemetry.go Implements rejection and recovery counters and structured logs.
pkg/​throttle/​safety_telemetry_test.go Tests attribution, exclusions, log suppression, redaction, and recovery transitions.
pkg/​throttle/​check.go Records MySQL rejections and identifies specific rejection reasons.
pkg/​throttle/​check_result.go Adds internal rejection metadata without changing JSON output.
doc/​mysql.md Documents metrics, log behavior, and collection limitations.

Review effort: Balanced (adaptive)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants