Skip to content

feat(qwp): keep idle WebSocket connections alive with PINGs - #109

Open
jerrinot wants to merge 7 commits into
mainfrom
jh_idle_pings
Open

jerrinot wants to merge 7 commits into
mainfrom
jh_idle_pings

Conversation

@jerrinot

@jerrinot jerrinot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Idle QWP connections get closed by load balancers, proxies, firewalls, NATs and the server's own idle timeout. The sender reconnects without data loss, but it's noisy and the log wrongly suggests data is being re-sent.

  • New keepalive_interval_millis: after this long with nothing sent, the sender pings. Default 30 s (on by default), 0 disables.
  • durable_ack_keepalive_interval_millis=0 now disables only the durable-ack PING. To stop all PINGs, also set keepalive_interval_millis=0.
  • PINGs use the same send timeout as data frames instead of 1 s, so a slow server or a GC pause no longer forces a reconnect and replay.
  • Reconnect log no longer claims a replay when nothing is re-sent: replaying from FSN 2 → resuming at FSN 3, 0 unacked frames to replay.

Server side: kept-alive idle connections now hold their WAL writers indefinitely instead of releasing them when the 5-minute idle reaper closes the connection. Tracked in questdb/questdb#7931.

The reconnect log said "replaying from FSN N", where N was fsnAtZero:
the next unacked FSN minus any symbol-dictionary catch-up frames. On an
idle connection dropped by a load balancer it read like a replay even
though nothing was re-sent, and N pointed at an already-acked frame.

Log the first unacked FSN and the number of frames actually replayed.
Load balancers, proxies, firewalls, NAT gateways and the server's own
idle timeout (http.net.connection.timeout, 5 min) close quiet QWP
connections, so idle senders reconnected over and over. Before this,
the I/O loop only pinged in durable-ack mode while acks were pending.

Add keepalive_interval_millis (default 30s, 0 disables): the I/O loop
pings once nothing has gone out for that long. It shares the throttle
and send path with the durable-ack keepalive.
…bilityTest

Only Sender and its builder are public API. The test pinned internal
signatures (QwpWebSocketSender factories, CursorWebSocketSendLoop
constructors) whose only callers live in this jar, so they can never
link against a different version. Widen the existing factory in place
instead of adding an overload.
@jerrinot

Copy link
Copy Markdown
Contributor Author

Verdict: approve. Level-3 review of head ebb6a755b5023a1ce3b14093b990fae1a2bac286 against base 70a7f853ccb81a5de94f0b128cb5174e7394a635: no Critical, Moderate, or Minor findings. Correctness and test gates pass; zero admitted coverage gaps.

  • Local JDK 8: 403 tests, zero failures/errors, one skip.
  • GitHub JDK 8 run: 3,594 tests, zero failures/errors, nine skips. JDK 25/26 checks passed.
  • Mutation checks caught missing idle PINGs and broken disabling. Pool, dictionary recycling, and reconnect probes passed.

Azure coverage build 279451 failed in ServerErrorAckTerminalTest.testServerErrorAckIsTerminalAndDoesNotBurnReconnectBudget. Its first sender.flush() is unguarded: the intentional server rejection can arrive before that call's trailing error check. A scratch-only 50 ms scheduling pause immediately after frame publication reproduced the same PARSE_ERROR on both base and head, with the test unchanged. Without the pause, 100 repetitions passed on each revision. This establishes an existing test timing assumption; the actual CI scheduling/JIT trigger remains unknown. The CI failure occurred after 13 ms, before the default 30-second idle PING could fire. The tested CI merge has the same tree as the reviewed head.

No matching OSS/Enterprise tandem exists; the new scheduler was not exercised against a real server during this review. Existing server PING handling and tests were inspected. No submodule changes; the primary working tree was unchanged.

@jerrinot

Copy link
Copy Markdown
Contributor Author

The CI failure in ServerErrorAckTerminalTest (Coverage Report job) is an existing race in that test, unrelated to this PR. Tracked in #110.

pendingDurable is emptied on reconnect and refills only after a frame
is sent on the new connection, which stamps lastFrameOrPingNanos, so the
reset could never make the durable-ack PING fire sooner.
A PING timeout reconnects and replays unacked frames. With 1 s, a GC
pause or a slow-draining server could tear down a live connection.
@mtopolnik

Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 29 / 31 (93.55%)

file detail

path covered line new line coverage
🔵 io/questdb/client/Sender.java 11 13 84.62%
🔵 io/questdb/client/cutlass/http/client/WebSocketClient.java 1 1 100.00%
🔵 io/questdb/client/impl/ConfigSchema.java 1 1 100.00%
🔵 io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java 3 3 100.00%
🔵 io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java 13 13 100.00%

@jerrinot

Copy link
Copy Markdown
Contributor Author

Verdict: approve. Level-3 review of head 31f9e41f88d5a546727f71a18e22a051c41e3a1a against base 70a7f853ccb81a5de94f0b128cb5174e7394a635.

No Critical, Moderate, or Minor findings. Correctness and test gates pass; zero admitted coverage gaps.

  • Local JDK 8: 553 tests, zero failures/errors, one skip.
  • Runtime probes: pool/recycle behavior passed; mutation checks caught broken PING scheduling, disabling, throttling, and config forwarding.
  • Timeout change: a controlled 1.2-second stall succeeds on head with a 5-second transport budget; base's fixed one-second PING timeout fails.
  • Exact-head CI: 3,594 unit tests, zero failures/errors, nine skips. JDK 25/26 checks passed.

API scope follows the PR's explicit policy that the changed low-level classes are internal.

Limitations: no matching OSS/Enterprise tandem or real-server/HA execution. At posting time, Azure's Windows job remains pending; no failed checks are reported.

No submodule changes. Primary checkouts unchanged.

@jerrinot

Copy link
Copy Markdown
Contributor Author

Verdict: approve. Level-3 review of head 31f9e41f88d5a546727f71a18e22a051c41e3a1a against base 70a7f853ccb81a5de94f0b128cb5174e7394a635. No Critical or Moderate findings, one Minor. Correctness and test gates pass; zero admitted coverage gaps.

Minor

PR description leaves out two behaviour changes

  • Problem: the body doesn't mention two changes existing users will notice.
  • Net impact: release notes understate what changes on upgrade.
  • Evidence: PR body vs. commits 47a4322b and 31f9e41f.
    1. durable_ack_keepalive_interval_millis=0 no longer disables all PINGs. Users who set it to stop PINGs now get an idle PING every 30 s unless they also set keepalive_interval_millis=0.
    2. The PING send timeout went from 1 s to the WebSocket client's default, which is 600 s.
  • Fix: add one line for each to the body, since the squash commit becomes the release note.

Trade-off (not a finding)

The PING timeout is 600 s, not 30 s: the WebSocket client is built with DefaultHttpClientConfiguration (QwpWebSocketSender.newWebSocketClient), whose getTimeout() returns 600_000.

  • Base: a PING to a peer that stopped reading, with a full send buffer, timed out after 1 s and failed over. This was most likely with the 200 ms durable-ack PING.
  • Head: that case can now hold the I/O loop for up to 10 min. Data frames already behave this way on base.
  • Why it's acceptable: the change is deliberate, because the 1 s timeout tore down healthy but slow connections. close() still breaks a blocked PING after its graceful window via closeTraffic().

What was checked

  • Server compatibility: every QWP server release since 9.4.0 handles PING on ingress (flushes pending acks, then sends a PONG).
  • Store-and-forward invariants: a failed PING goes through fail(), the normal reconnect path. It adds no poison strike and cannot reach the startup terminal, because the loop body runs only after swapClient.
  • Removed throttle reset (03b7fc70): dead code, as claimed. pendingDurable is cleared on the same reconnect and refills only after a send, which stamps the clock.
  • Config: the view-based path (the one ws::/wss:: configs use), the builder guard, the setter wiring before start(), toMap and ConfigSchema are all correct. Values of 0 or below disable the PING; very large values saturate without overflow.
  • API: the deleted ExportedApiCompatibilityTest, restored and run against head, still passes (6/6). No caller of the widened connectWithCredentialSupplier or of sendPing exists in questdb or questdb-enterprise.
  • Java 8: no Java 9+ APIs or syntax were added.
  • Mutation spot-check: tests catch broken wiring, a broken throttle, broken disabling, broken view-path parsing, and a PING that costs a reconnect. The surviving mutants are low-risk: the 1 s/600 s timeout, the start-of-loop clock, and the reconnect log text.

Coverage and scope

  • Tandem: no OSS or Enterprise tandem PR exists (searched by branch jh_idle_pings). If the server did not treat a PING as activity, idle connections would just be reaped as they are on base, so nothing gets worse.
  • Submodules: no pointer changes.
  • Severity: 0 Critical / 0 Moderate / 1 Minor.

@jerrinot jerrinot added the READY label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants