fix(TCrossHttpClient): retry once on stale reused keep-alive connection

[freitasjca]

Problem

TCrossHttpClient surfaces a 400 to the caller when a reused pooled connection goes stale between requests. The stale keep-alive race is inherent in connection pooling: the server closes the connection; the client's write still succeeds locally (TCP send buffer), then the next read returns an error. Every mainstream HTTP client — WinHTTP, curl, Go net/http — retries transparently.

Solution

In TCrossHttpClientConnection.TriggerResponseFailed, detect the stale-connection case and re-dispatch the request once through TServerDock.DoRequest on a fresh connection.

Gates (all required):

• Not in Destroy teardown (FNoRetry) — avoids touching freed dock state
• Not already retried (FRetried) — bounded to one retry per request
• Body replayable: pointer+size or none (TBytes/TCustomMemoryStream qualify; chunk-source TStream closures do not)
• AND one of:
  • (a) dispatch rode a reused idle connection (FViaReusedConnection) and zero response bytes received (FRespDataReceived=False) — stale keep-alive race, safe for any method including POST
  • (b) method is idempotent (GET/HEAD/PUT/DELETE/OPTIONS) — RFC 7230 §6.3.1 explicitly permits automatic retry

Design choices:

• Forces a fresh connect on retry (TServerDock.DoRequest skips GetIdleConnection when FRetried): under concurrent bursts every idle pooled connection goes stale together, so retrying on another idle connection fails again.
• 50 ms pause on a throwaway thread — never blocks the IO thread; an immediate retry dies in the same backend transient that killed the first attempt.

Testing

Validated against fphttpserver (which closes keep-alive connections as a group on server-side restart/reload), eliminating flaky failures on HEAD (test 07) and concurrent requests (tests 18 and 29) in a 101-check suite run with TCrossHttpClient.

[winddriver]

Thank you for the contribution and for investigating stale keep-alive failures. After reviewing commit dd668e4a271f0491f0cb5869093e67b7fd4fdf2a, we are closing this PR without merging because the proposed automatic retry introduces correctness and lifecycle risks in the generic HTTP client.

1. Zero response bytes do not prove that the server did not execute the request. The condition at Net.CrossHttpClient.pas:1988–1991 permits replaying POST/PATCH on a reused connection. A server can commit a business operation and then lose the connection before any response reaches the client. Retrying would execute that operation twice. RFC 9110 §9.2.2 describes this heuristic as a riskier approach; it does not establish that replay is safe. A replayable request body does not guarantee business idempotency.

2. The delayed retry is outside cancellation and destruction management. The anonymous thread at lines 2069–2074 holds the request during a 50 ms sleep, when it is no longer in the connection or pending-request queue. CancelAll cannot cancel that work, so it can send a request after cancellation. Retaining IServerDock keeps the dock alive, but its FClientSocket is a raw object reference; the worker is not joined during client destruction, creating a potential use-after-free window. FNoRetry does not cancel workers that have already started.

3. Forcing a fresh connection can leave a retry queued despite a healthy idle connection. At lines 4034–4035, retries skip idle connections but still obey the connection-count limit. With MaxConnsPerServer=1, request A can fail and schedule its delayed retry, while queued request B opens a replacement connection and completes before A wakes up. A then refuses that healthy idle connection, cannot open another, and is queued. With Idleout=0 and no further requests or peer disconnect, there may be no event to advance A or deliver its completion callback.

4. The failure hook is broader than a stale-connection detector. TriggerResponseFailed also handles parsing and compression failures. The idempotent-method branch does not classify the failure, so it also retries errors that are not transient transport failures.

5. The new console diagnostics expose complete URLs in release builds. The direct Writeln calls at lines 2000–2006 and 2049–2052 can disclose query-string tokens or signatures and bypass the library's logging controls.

These findings are based on static code review; the concurrency scenarios have not been reproduced in a runtime test here. The reported 101-check run is useful context, but the PR does not include those tests or regression coverage for the cases above.

For this library, retry policy should remain under caller control. Any future retry feature should be explicitly enabled, distinguish eligible transport failures, establish request replay safety, and use scheduling owned by the client that participates in cancellation, shutdown, and connection-pool limits. Those requirements need a different design, so we will not merge this implementation.