Login path saturates at ~550-700 connects/s per instance regardless of cores (production report)

[ryderhwang]

Summary

In a large production deployment we've measured PgDog's client login path saturating at ~550–700 new client connections/s per instance, while the instance uses only ~0.5 of its 4 cores (workers = 4). Past that rate, TCP accepts succeed but startup/auth lags; clients hit their own connect_timeout mid-startup and PgDog logs and reports nothing (no errors, no warnings), because from its perspective the connection simply went away before login completed. Adding instances fixes it linearly; adding cores to an instance does not — which suggests a serialization point in the login path rather than CPU exhaustion.

Environment

• pgdog v0.1.52, pooler_mode = "transaction", passthrough_auth = "enabled_plain_allow_change", no client TLS
• 4 vCPU / workers = 4 per instance, Kubernetes (arm64), listen backlog raised via sysctls (accept-queue overflow was measured and ruled out as the main cause — ~373 overflow drops vs ~130k client-side connect timeouts during the worst event)
• Client fleet: Django/psycopg2, ~180 app instances, a fresh connection per request (no client-side pooling), so a rolling deploy of the app fleet produces reconnect waves of ~7k connects/s fleet-wide

What we observed

• At 8 instances (~600/s/pod capacity each), a fleet-wide reconnect wave (~7k/s) drove client-side connect p99 to a pinned 5s (their libpq connect_timeout), with tens of thousands of failures over ~30 min — while pgdog.errors stayed 0, CPU stayed ~12% of the pod, and logs were silent.
• Scaling to 32 and then 80 instances made the same waves invisible (~175/s/pod → connect p99 ~20ms flat).
• Separate load testing measured the per-instance wall at ~700 connects/s against a warm instance and ~550/s under sustained load, roughly independent of core count.

Reading the code, candidate suspects

(From frontend/listener.rs / frontend/client/mod.rs / backend/databases.rs on main — happy to be corrected.)

1. With passthrough auth, every login calls databases::add(user). The password-matches path looks cheap (read + constant_time_eq), but the password-differs / first-seen path takes the global config lock, clones the entire config, calls set() and reload_from_existing() — per login. During a credential rotation with allow_change, a reconnect wave would hit this path on every login. We don't believe this is our steady-state wall (our password is stable), but it looks pathological under rotation + storm.
2. The login exchange is a sequential multi-await conversation (extra ClearTextPassword round trip under passthrough), so logins-in-flight can pile up — but that alone shouldn't cap throughput at ~600/s on an idle CPU.
3. Everything spawned from a single accept task via comms.tracker().spawn() — accept itself seems unlikely to be the wall at these rates.

Ask

• Is a ~550–700 logins/s per-instance ceiling (independent of workers) expected? Is there a known lock/serialization point in the login path?
• We're happy to profile a build under a synthetic connect storm and contribute a fix if you can point us in the right direction — same deployment that contributed feat: configurable listener backlog to allow more incoming connection requests #1452.

Related: we'll file a separate issue about making login-path saturation observable (today it's invisible in both logs and metrics).

[levkk]

Possibly related #1468

[ryderhwang]

Closing this with data: we benchmarked the login path directly and cannot reproduce any userspace ceiling remotely near the ~550–700/s we reported. The bottleneck we hit in production is not in PgDog.

Method

A purpose-built harness performs complete logins in a loop across N concurrent workers — TCP connect → StartupMessage → AuthenticationCleartextPassword exchange → wait for ReadyForQuery → close — with SO_LINGER=0 (RST close) so the load generator never accumulates client-side TIME_WAIT and can sustain arbitrary connect rates. PgDog configured as in our production: pooler_mode = "transaction", passthrough_auth = "enabled_plain_allow_change", no client TLS, real Postgres 14 behind it.

Results

Setup	logins/s
Linux x86_64, pgdog pinned to 4 cores (taskset), workers = 4, main	~78,000
Linux, same, concurrency 64 → 1024	47k → 78k → 75k (flat, no cliff)
macOS arm64, workers = 4, main	~25,000–26,000
macOS arm64, workers = 4, v0.1.52 (our production pin)	~25,000–26,000 (identical)
macOS arm64, workers = 1	~15,000
Alternating passwords every other login (allow_change rotation storm), workers = 4	~24,000 (20 transient errors / 193k logins — the only mild finding)

Client-observed avg login latency at moderate concurrency: 0.6–3ms; zero errors in the steady-state runs.

What we got wrong, for the record

• Our "~550–700 connects/s/pod" number is uncannily close to a load-generator artifact: a single-source-IP tester doing graceful closes caps at roughly ephemeral_port_range / TIME_WAIT ≈ 28,231/60s ≈ 470–700/s regardless of the server under test. Our harness only breaks past it because it RST-closes.
• The production incident signature (TCP "connected", client times out mid-startup, PgDog logs and metrics silent) is fully explained by in-front-of-PgDog drops: with the pre-feat: configurable listener backlog to allow more incoming connection requests #1452 hardcoded backlog of 1024, an overflowing accept queue on Linux (default tcp_abort_on_overflow=0) drops the handshake-completing ACK — the client sees ESTABLISHED and waits for an auth response on a connection the proxy never received. Conntrack pressure under NAT'd Kubernetes service routing acts the same way. Neither is visible to PgDog userspace, which is precisely why nothing was logged.
• The databases::add full-config-clone path under allow_change (my earlier suspect Override user database credentials #1) is real but benign in practice — the reload diffs and reuses pools, so even a worst-case alternating-password storm barely dents throughput.

Follow-ups that still stand

• feat: configurable listener backlog to allow more incoming connection requests #1452 (configurable listen_backlog) is the actual mitigation for the accept-queue failure mode — thanks again for merging.
• Add login path observability: login_duration histogram, in-flight gauge, abandoned counter #1471 adds login-path observability. Worth noting a limitation this investigation clarified: logins_in_flight/login_duration see only connections that PgDog accepted — kernel-level pre-accept drops need TcpExtListenOverflows/conntrack counters alongside. (During validation, the metrics tracked the harness exactly: 119,068 logins counted on both sides at 15k logins/s.)

Apologies for the misdirected report, and hopefully the numbers are useful as a reference point: the login path comfortably sustains tens of thousands of logins/s per instance. If we ever observe a genuine userspace wall in production telemetry we'll reopen with a reproducer.