Add login path observability: login_duration histogram, in-flight gauge, abandoned counter - #1471
Add login path observability: login_duration histogram, in-flight gauge, abandoned counter#1471ryderhwang wants to merge 3 commits into
Conversation
When the login path saturates, TCP accepts succeed but clients time out mid-startup and PgDog reports nothing: no error, no log line, no metric. Make that failure mode observable on the OpenMetrics endpoint: - logins_in_flight: connections accepted but not yet ReadyForQuery, the leading indicator of login-path saturation - logins: completed client logins - logins_abandoned: clients that started the Postgres handshake but disconnected (or timed out) before login completed; connections that close without sending a single startup byte (load balancer TCP health checks) are excluded - login_duration: accept-to-ReadyForQuery histogram (ms) for completed logins A LoginTimer guard is created at accept, engaged on the first startup message, marked successful when login completes, and disarmed when PgDog ends the login with an explicit response (auth failure, shutdown, pool down) or the connection is a cancel request. Dropping it engaged but unfinished counts an abandoned login, so every exit path is covered without touching each error branch. The scalar metrics also flow through the OTLP exporter; the histogram is rendered only on the OpenMetrics text endpoint via a new overridable OpenMetric::render_measurements (the OTLP renderer currently only speaks gauge/sum). Closes pgdogdev#1470 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Validated under load while benchmarking for #1469: with the harness driving 15,000 logins/s through a single instance ( One scope note from that investigation: these metrics see only connections PgDog has accepted. Kernel-level pre-accept drops (accept-queue overflow, conntrack) remain invisible by nature — |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The multiline writeln invocations left llvm-cov region artifacts on their continuation lines; captured-identifier format strings keep each write on one line and read cleaner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Re the codecov report — the 3 uncovered patch lines are all in paths that have no coverage on
Happy to add a raw-socket GSSENC test or factor the exporter's collection step out into a testable fn if you'd like those covered, but I didn't want to restructure existing code for it in this PR. |
Implements #1470 (context in #1469).
What
Four new series on the OpenMetrics endpoint:
logins_in_flight(gauge) — accepted but not yet ReadyForQuery; the leading indicator of login-path saturationlogins(counter) — completed loginslogins_abandoned(counter) — clients that started the Postgres handshake but went away (or hitclient_login_timeout) before login completedlogin_duration(histogram, ms) — accept → ReadyForQuery latency of completed loginsDesign
LoginTimerRAII guard is created at accept, engaged on the first startup message, success-marked whenClient::spawngets a logged-in client, and disarmed when PgDog ends the login with an explicit response (auth failure, shutting down, pool down) or the connection is a cancel request. Dropping it engaged-but-unfinished counts one abandoned login — every exit path is covered without instrumenting each error branch.logins_abandoned— those never engage the guard.OpenMetric::render_measurements(default keeps today's behavior for every existing metric), emitting standard_bucket/_sum/_countwith cumulative buckets. Buckets are 1ms–30s; happy to change bounds.Verification
cargo nextest run --profile devshows an identical pass/fail set to currentmainon the same machine (all pre-existing/env failures, zero deltas).passthrough_auth = "enabled_plain_allow_change", transaction mode: a real psql login renderslogins 1+ one histogram observation; a bare TCP connect/close renders nothing; a client that sends StartupMessage, receives AuthenticationCleartextPassword, and closes renderslogins_abandoned 1.cargo fmt+cargo clippyclean.Why
During our worst production login-path saturation event (~130k client-side connect timeouts over ~30 min, details in #1469), every PgDog-side signal was green — TCP accepted, clients gave up mid-startup, and nothing was logged or counted. With these metrics the event is a one-glance diagnosis:
logins_in_flightclimbing withlogin_durationp99 stretching toward the clients' connect timeout, thenlogins_abandonedticking up.