Socket Mode: reconnects behind NAT leak server-side connection registrations → too_many_websockets cap → silent loss of interactive payloads; mitigation proposals #1940

Activity

1. lubosxyz commented on Aug 13, 2026

   

   lubosxyz

   on Aug 13, 2026

   Author

   More actions

   Follow-up with measured data after deploying client-side mitigations (the wire
   instrumentation from the report, plus an exponential-backoff idle watchdog:
   900 -> 1800 -> 3600 s cap, +-15% jitter, reset only on a delivered envelope).

   One night before vs. one night after, same app, same host:

   metric (per night)	before	after
   idle-watchdog reconnects	73	4
   hello num_connections	climbed to 10 (cap)	1 on every hello
   too_many_websockets bursts	2 (11 frames)	0

   Two observations that may help triage:

   1. Reducing the reconnect rate keeps the primary app's pool clean, which
      supports the accumulation model (ghosts are minted by reconnects whose close
      never reaches the server).
   2. However, other single-instance apps on the same host still sit at a steady
      num_connections of 4-5 overnight (network-level flaps still force
      occasional SDK reconnects, and each ghost lingers for the ~5 h
      approximate_connection_time horizon). Client-side tuning cannot get below
      that floor — which is why surfacing num_connections (proposal 1/2) and a
      server-side eviction or force-disconnect API would still be valuable even
      for well-behaved clients.
2. WilliamBergamin commented on Sep 3, 2026

   

   WilliamBergamin

   on Sep 3, 2026

   Last edited by WilliamBergamin

   Contributor

   More actions

   Hi @lubosxyz thanks for the exceptionally detailed report 💯

   the wire-level instrumentation and the before/after num_connections data make this failure mode very clear.

   Status on referenced issues: both are now merged and released in 3.44.1:

   • fix(socket-mode): only release connect_operation_lock when this task acquired it #1926
   • fix: stop aiohttp Socket Mode connect() retrying forever after close #1956

   Both reduce the spurious/concurrent reconnects that are one of the mechanisms minting the ghost registrations you describe, so they should help at the margins.

   That said, I don't think these resolve this issue, so I'd like to keep it open. As you framed it, this is the server-side counterpart: the root cause is registrations aging out on Slack's side over ~5 h, which the SDK might not fix directly. Your follow-up data makes the decisive point, after cutting reconnects 73 → 4, other well-behaved single-instance apps still sat at num_connections 4–5 overnight, so reconnect-correctness fixes alone can't get below that floor.

   I'm narrowing the scope of this issue to the SDK-actionable proposals that remain unaddressed:

   1. Surface hello metadata (num_connections, approximate_connection_time) and disconnect reasons. Maybe via a public callback and/or INFO logging.
   2. Warn when hello's num_connections exceeds a threshold while the client holds fewer connections.
   3. make-before-break reconnect with close-confirmation, or at minimum documenting the NAT hazard.
3. WilliamBergamin

   added

   discussionM-T: An issue where more input is needed to reach a decisionM-T: An issue where more input is needed to reach a decision

   auto-triage-skip

   on Sep 3, 2026
4. gorkem2020 commented on Oct 14, 2026

   

   gorkem2020

   on Oct 14, 2026

   More actions

   A data point from a Node deployment that may help with the server-side part: one event delivery clears the dead registrations at once.

   Setup: @slack/socket-mode 3.0.1 (via OpenClaw's Slack plugin), five single-instance apps behind one gateway. The gateway holds exactly one ESTABLISHED socket per app (checked host-wide, containers included). To read the count we open a short-lived probe connection, read num_connections from hello, and close it cleanly; the probe never acks anything.

   • After a network outage on our side, all five apps hit Slack's cap. Nine hours later they still sat at 8 or 9 registrations each while the gateway held one. A single DM to a bot (one events_api delivery) took its app back to 1, just our connection, within seconds. We did this for four apps. The app we did not message kept its extras and only lost about one every few minutes.
   • A bot-originated event works too. The bot called reactions.add, then reactions.remove, on one of its own old messages, which produced one reaction_added and one reaction_removed delivery. That took the fifth app from 5 registrations to 1. A Web API call that delivers no event (auth.test) changed nothing.
   • On fresh test apps (Socket Mode only, never installed) we could not produce a lasting registration from the client side. We tried a clean close, FIN without a close frame, RST, a close frame plus FIN, a connection blackholed in both directions, a handshake whose 101 and hello never reached the client, unused apps.connections.open tickets, a refused 11th connection (too_many_websockets), and full client network blackouts of 30, 90 and 180 s. Every case was back at baseline within 61 s: Slack pings every 10 s and drops a connection after about 4 missed pongs.

   So the registrations that pile up look like entries Slack only re-checks when it tries to deliver an event to them. An idle app keeps them for hours, and its routine 5-hour refresh then hits too_many_websockets.

   App-side mitigation we now run: whenever hello reports more connections than the client holds, or Slack refuses a connection with too_many_websockets, the bot sends itself one event (a reaction add and remove on an old bot message). It needs reactions:write.
5. WilliamBergamin commented on Oct 14, 2026

   

   WilliamBergamin

   on Oct 14, 2026

   Contributor

   More actions

   I've been made aware that a server side patch for this should be going out soon 💯 once confirmed we should be able to close this issue thanks for reporting this 🙏
6. WilliamBergamin commented on Oct 21, 2026

   

   WilliamBergamin

   on Oct 21, 2026

   Contributor

   More actions

   Hi @lubosxyz @gorkem2020 👋 the server-side fix for this has now rolled out to production 🚀

   Your data was key to tracking it down, especially the observation that a single event delivery clears the extra registrations, and that none of the client-side close/blackhole scenarios could produce a lasting one. That pointed us to the root cause: under certain server-side conditions, Slack could re-register a connection entry for an app on a server that no longer held any of its connections. Those stale entries counted toward num_connections and the 10-connection cap, and they were only cleaned up when an event delivery attempted to reach them. That fits the "one DM clears it" behavior you saw.

   With the fix, hello should report num_connections that matches the connections your client actually holds, including after network flaps and reconnect-heavy periods. You should no longer need workarounds like the self-event mitigation.

   I'm going to close this one out. If you still see num_connections climb above what your client holds, please reopen with the hello / disconnect logs and we'll dig back in 🙏

   Thanks again for the exceptional reports, they made this fixable 💯
7. WilliamBergamin

   closed this as completedon Oct 21, 2026
8. gorkem2020 commented on Oct 28, 2026

   

   gorkem2020

   on Oct 28, 2026

   More actions

   Follow-up after the server-side fix: from 2026-10-19 to 2026-10-26 we saw no extra registrations at all (five Socket Mode apps, one connection each). Since then the hello frame has reported more connections than we hold three times, each right after Slack's routine refresh_requested disconnect:

   • 2026-10-26 23:15 UTC, app A: num_connections: 2
   • 2026-10-27 04:15 UTC, app A: num_connections: 2
   • 2026-10-28 00:23 UTC, app B: num_connections: 3

   At each point our process held exactly one socket per app, app A's later refreshes came back with num_connections: 1, and there were no too_many_websockets refusals. Is a short-lived count like this expected around refresh_requested, or does it point to the same registration issue? Happy to share more timing detail.