Security/robustness hardening: handshake verification, message-size bounds, TLS IP SANs, mask entropy, liveness

[voicetel]

This PR adds a set of security/robustness hardening fixes to the WebSocket
client transport, found during a code review of a downstream integration.
Each fix is an independent commit.

The client treats the server side as potentially hostile (untrusted input),
so several of these close remote DoS / integrity gaps:

• Verify Sec-WebSocket-Accept against base64(SHA1(key + GUID)). The
  handshake previously accepted on mere presence of the header (RFC 6455 §4.1
  requires matching the computed value); the computed accept was dead code.
• Bound permessage-deflate output — rxInflate appended inflate output
  with no cap, so a small compressed frame could expand without limit
  (decompression bomb → OOM). Capped at a 16 MiB message ceiling.
• Bound single-frame and reassembled message size — no maximum existed on
  a peer-declared 64-bit frame length or on fragments accumulated across
  continuation frames. Reject with close code 1009.
• Bound the pre-upgrade handshake response — a server could stream header
  bytes forever without the terminating CRLFCRLF, growing the buffer and
  re-scanning it each read. Fail past 64 KiB.
• Seed the frame-mask PRNG from a strong entropy source — the per-frame
  masking key was seeded from time() + a stack address (predictable); RFC
  6455 §5.3 requires unpredictable masks. Now seeded from std::random_device
  (as the handshake nonce already is), keeping per-frame masking syscall-free.
• Reject RSV1 on control/continuation frames — RFC 7692 §6 permits the
  compression bit only on the first frame of a message.
• Verify IP-literal TLS endpoints against iPAddress SANs — use
  X509_VERIFY_PARAM_set1_ip_asc for IP hosts instead of set1_host (which
  matches dNSName SANs).
• Strip CR/LF from configured URI/host/headers in the handshake request to
  prevent header/request injection from local configuration.
• Detect a half-open peer via ping/pong liveness — heartbeat pings were
  fire-and-forget and PING_TIMEOUT was unused; now the connection is closed
  after N unanswered pings. Gated on ping_interval > 0, so behavior is
  unchanged when the heartbeat is disabled.

Independent of, and non-conflicting with, the _ctx thread-safety PR (#6);
this branch is based on main and touches only WebSocketContext.* and
WebSocketReceiver.*.

[amigniter]

Thanks for the thorough review and for splitting the fixes into independent commits. I went through them individually.

Most of the findings look valid and useful, and I am happy with the general direction of the PR.

The following changes look good to me:

• Sec-WebSocket-Accept verification — valid fix. The client was computing the expected accept value but was not actually comparing it with the server response.
• permessage-deflate output limit — valid protection against unbounded inflate output / decompression bombs.
• single-frame and fragmented-message size limits — valid and complementary to the inflate limit. Using close code 1009 makes sense here.
• RSV1 validation on control/continuation frames — valid protocol fix.
• IP-literal TLS verification using X509_VERIFY_PARAM_set1_ip_asc() — valid fix for certificates using iPAddress SANs.
  -ping/pong liveness handling — useful robustness improvement. I do not consider this one primarily a security issue, but the behavior itself makes sense.

There are a few changes I would like to adjust before merging.

Handshake response size limit

The 64 KiB handshake limit is a good idea, but I think the current check happens too late.

The code first does something equivalent to:

const size_t len = evbuffer_get_length(input);

std::vector snap(len);
evbuffer_copyout(input, snap.data(), len);

and only afterwards checks whether the HTTP header terminator was found and whether the accumulated handshake exceeds MAX_HANDSHAKE_BYTES.

This means a hostile peer can already cause a large allocation before the limit is applied.

At the same time, simply rejecting whenever the complete evbuffer is larger than 64 KiB would not be correct either, because the buffer may already contain a small valid HTTP upgrade response followed by WebSocket frame data.

I would prefer to search for \r\n\r\n only within the first MAX_HANDSHAKE_BYTES bytes, ideally directly using the evbuffer API, and:

terminator found within the limit -> process handshake

no terminator and buffered handshake > limit -> fail

otherwise -> wait for more data

That would enforce the limit before copying/allocating based on an attacker-controlled buffer size.

Frame masking PRNG

I agree that time() plus an address is not an appropriate seed for WebSocket masking.

However, I would like to look more carefully at the proposed approach of using std::random_device only to seed the existing PRNG.

RFC 6455 requires each masking key to be unpredictable, including that observing previous masking keys should not make subsequent keys predictable.

If the existing next_u32() generator is a small non-cryptographic PRNG, giving it a strong initial seed may still not satisfy that property once outputs are observable by the peer.

I would therefore prefer either:

• generating masking keys from a cryptographically secure source, or

• using a cryptographically suitable PRNG seeded from strong entropy.

I would like to avoid relying only on a secure seed if the generator itself is predictable from its outputs.

CR/LF sanitization

I agree with the request/header injection issue, but I am not sure silently stripping CR/LF/NUL is the best API behavior.

For example, turning:

abc\r\ndef

into:

abcdef

creates a value the caller did not actually provide.

I would prefer rejecting invalid URI/host/header input when it contains forbidden CR/LF/NUL characters rather than modifying it silently.

Header names may also deserve stricter validation than just CR/LF stripping.

Sec-WebSocket-Accept parsing

The actual accept-value verification is definitely needed.

One minor improvement I would suggest is avoiding a substring search for:

sec-websocket-accept:

across the complete HTTP response.

It would be more robust to parse individual HTTP header lines and compare the header name case-insensitively while keeping the Base64 value case-sensitive.

This is not a blocker for the security fix itself, but it would make the handshake parser more precise.

Overall, thanks for the work on this. The majority of the findings are real issues and I would like to merge them. I mainly want to tighten the handshake-size handling, masking-key generation, and input-validation behavior before taking the complete PR.

Best,
/m