fix(core): serve a connection that arrives while the listeners are paused, and count the accepts the cap refuses (#359)

CHANGELOG.md

@@ -19,6 +19,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **A connection that arrived while the listeners were paused was answered 503 instead of waiting in the backlog (#359).** The pause at the connection cap and under CoDel calls `stop()` on each listen event, which in php-async did not stop accepting (true-async/php-async#305): the connection reached the accept callback and the hard-cap safety net refused it. `h1/062`, a third client during a pause at `setMaxConnections(2)`, read `503 Service Unavailable` on 5 of 5 runs; with php-async's fix it reads 200 after the resume, 10 of 10, as do `h1/063` (three connections at once past a cap of two) and `h1/064` (a resume that pauses again while delivering). Needs php-async with #305. `getTelemetry()` and `getStats()` gain `accepts_refused_at_cap_total`, which a pause that holds keeps at 0, and `accept_errors_total`, for accepts the reactor reported failed and the server used to drop without a trace. Under `SO_REUSEPORT` a paused worker still receives its share of new connections, which wait for it.
+
 - **After `stop()` a connection kept serving, into a server scope `start()` had already cancelled (#345).** `stop()` released the listeners and left the connections open, and nothing that starts a request checked whether the server still ran. A keep-alive connection answered a request sent after `start()` returned, and a request in flight answered every request pipelined behind it, without `Connection: close`. Once the cancelled scope was empty php-async freed it, and the next request built its scope from freed memory: with 150 GETs pipelined behind a stuck handler, 3 to 4 runs of 30 ended in SIGSEGV. `stop()` now retires every connection: one with nothing in flight closes at once, one running a handler finishes that response with `Connection: close` and closes, the requests behind it are not run, and HTTP/2 sends GOAWAY and refuses streams opened before the peer read it. A server started again drains its handlers at the next `stop()` too; the drain ran once per object, so the second `start()` returned with its handler still running. Evidence: `core/081`, `core/082`, `h2/070` and `tls/019` fail against `48fa65b` and pass here.
 - **Pipelined HTTP/1 requests past the 8 KiB read buffer closed the connection while a handler was busy (#341).** The requests waiting behind the busy one filled the buffer, and the allocator's only answer to a full buffer was an empty one, which libuv reports as `UV_ENOBUFS` and the reactor as a read error: 104 of 151 pipelined requests were answered, 64 of 3001. The read now stops while the buffer is full, so the kernel and the TCP window hold the peer back, and starts again once the buffer is half empty; all 151 and all 3001 are answered, in order. A refusal behind a stopped read still drains the body, and frames pipelined behind a WebSocket upgrade still reach the handler. TLS reads through its own path and is unchanged.
 - **With the reactor pool on, nothing was compressed, an encoded request body reached the handler encoded, and `setJsonEncodeFlags()` was ignored (#350).** A worker builds the request and the response itself, and did not attach compression, decode the body or stamp the JSON default as the transports do; `sendFile()` on the pool also served identity where in-thread HTTP/3 served the `.gz` sidecar, because the reactor never saw the request's `Accept-Encoding`. Now a pooled response is encoded as in-thread HTTP/3 encodes it (buffered and streamed, with `Vary` and the encoded `Content-Length`), a `Content-Encoding` body is decoded or answered 415, 413 or 400, `json()` follows the configured flags, and `sendFile()` picks the sidecar. HTTP/3 without compression built in also lost the JSON flags, which it now keeps. Evidence: `h3/078` reads identity bodies, an undecoded POST and escaped slashes against `958e73e`, and passes here and with the pool off.

dev/PLAN.md

@@ -16,6 +16,65 @@ a test that fails without it, a measurement, or both. Issue numbers point at
   codec's own decoder, including through the `NEED_OUTPUT` loop; cost measured in
   `dev/BENCHMARKS.md`. Merged.
 
+## S1 — A paused listener leaves new connections in the backlog (item 12)  [in progress]
+
+Goal: `http_server_pause_listeners` stops accepting, so a connection that arrives
+during a pause is served after the resume instead of answered 503.
+Done when: `h1/062` passes on Linux, macOS and Windows CI against php-async `main`.
+Tests: first
+Base: 068cd09 (server), e580c15 (php-async)
+
+Cause: php-async's `libuv_listen_stop` cannot stop `uv_listen` (libuv has no such
+call), so accepts go on. Design E, by the Critic and the Sage on 2026-09-30,
+approved by Edmond: `stop` makes the connection callback return without
+`uv_accept`, and libuv stops watching the fd itself (`unix/stream.c:528-531`);
+the first `start` runs `uv_listen`, a later one unpauses and delivers the held
+connection on the next tick; Windows TCP listeners take one AcceptEx at a time,
+set before `uv_listen`. Rejected: a `uv_poll` accept loop (a busy loop on a
+socket several workers share on Windows, an EMFILE spin, no named pipes).
+
+- [x] S1.1 php-async: an asynctest cmocka test of stop and start, red first
+      done: stop, three connects, no callback; start, three callbacks on the next
+        tick and none inside start(); pause and resume twice with a held
+        connection; dispose while paused leaks nothing under ASan
+      tier: T2 · role: —
+      handoff: `sapi/asynctest/tests/test_listen_pause.c` in php-src (PR
+        true-async/php-src#38), 5 cases, all red on php-async e580c15. Under
+        ASan with LeakSanitizer (USE_ZEND_ALLOC=0): 5 of 5, 3 of 3 runs, no
+        report; a leak planted in listen_close_cb is reported (528 bytes).
+- [x] S1.2 php-async: design E in the listen event
+      done: S1.1 green; the php-async phpt suite passes; a callback that returns
+        without accepting while not paused is re-delivered, not lost
+      tier: T2 · role: Critic
+      Critic 2026-09-30 on the diff: single accept at creation slowed every
+        Windows listener (libuv can switch after listen) — moved to the first
+        stop(); the init-failure retry could spin — now a loud error; recover
+        skips a closing handle; a failed accept waits for the next tick. All
+        taken.
+      handoff: php-async PR #306 (commit 7965109). asynctest 5 of 5 green, 3 of
+        3 runs; php-async phpt 1214 of 1220, the 6 fail on main alike (a
+        symlinked ext/async breaks their ../../../../ includes).
+      Windows CI 2026-09-30: `h1/064` failed on #306. libuv keeps completed
+        AcceptEx requests in a LIFO stack, and the switch to single accept at
+        the first stop() completes only after all 32 pre-posted requests are
+        used, so a pause delivered the newest connection first. Sage: single
+        accept before the first uv_listen of every Windows TCP listener
+        (php-async PR #307); a FIFO queue in php-async was rejected. 064 green
+        on Windows with #307. Its accept-rate cost is item 33.
+- [x] S1.3 server: 062, a burst test and a re-pause test green on the patched
+        php-async; the comments at `http_server_class.c` on REUSEPORT and pause
+        say what happens; the 503 safety net and dropped accept errors are counted
+      done: the three phpt pass locally; `h1/062` fails on unpatched php-async
+      tier: T1 · role: —
+      handoff: 062, 063, 064 read 503 on php-async main and pass 10 of 10 on the
+        branch; the whole suite 516 of 516 on it. telemetry/009 lists the two
+        new counters. Issue #359.
+- [~] S1.4 Both PRs: php-async first, then the server against its `main`
+      done: both merged with every CI platform green; the Windows accept rate
+        before and after single accept is recorded in `dev/BENCHMARKS.md`, or
+        named as not measured
+      tier: T1 · role: —
+
 ## Next
 
 ### Order of the open defects, 2026-09-28
@@ -70,10 +129,9 @@ an item marked "reproduce" gets its failing run before any code.
     the suspect `proc_open()` pipe path is out of the suite. Closed if 070 does
     not hang again; the pipe hang itself, if real, belongs to php-async.
 11. HTTP/3 `chunk_queue` primer: with the next change to `h3_stream_append_chunk`.
-12. `http_server_pause_listeners` and the accept it may not stop: reproduce.
-    The accept callback also drops an error such as EMFILE with no log and no
-    counter (`http_server_class.c:2218-2223`); php-async re-arms the listener,
-    so only the signal is missing (health check 2026-09-30).
+12. `http_server_pause_listeners` and the accept it may not stop: stage S1
+    above (#359, true-async/php-async#305). The accept errors it used to drop
+    are counted in `accept_errors_total`; they are still not logged.
 12a. A failed or short HTTP/1 file body keeps the connection alive:
     `h1_static_on_static_done` and `h1_sendfile_on_done` discard `status`
     (`http_connection.c:2744`, `:2833`), the plaintext completion reports a
@@ -95,7 +153,11 @@ an item marked "reproduce" gets its failing run before any code.
     with six copies in parallel, on `main` and on the #313 branch alike; 8 of 8
     pass alone, and one full `-j4` suite of two hit it. A test that counts
     CPU should not see a bounded ring's designed drop; decide which side moves.
-14. `tas_free_port_span` on Windows: it hands out adjacent ports, and Windows
+14. The port helpers race. `tas_free_port` binds port 0, reads the number and
+    closes the socket, so another test's outgoing connection can take the port
+    before the test binds it: `core/027` failed its bind on 127.0.0.1:49353 on
+    macOS debug (run 36718896142, PR 360), passing on retry. Port 0 listeners
+    (81c0437) avoid the window. And `tas_free_port_span` on Windows: it hands out adjacent ports, and Windows
     gives ephemeral ports in order, so `base + 1` is the next port another test
     under `-j2` receives. `core/023` failed its bind that way (PR 337's Windows
     job) and PR 338 moved it and `core/079` to separate kernel-assigned ports;
@@ -166,6 +228,9 @@ Added by the health check of 2026-09-30 (`dev/HEALTH.md`), in this order:
     23 put the HTTP/3 and static targets in the population. The first run
     passes the known-answer check of rule 28.1; the survivors go into
     `dev/HEALTH.md` as the reference run.
+33. A connect storm on Windows CI (N parallel connects, time until all are
+    accepted), reported and not gating, before and after php-async #307: one
+    AcceptEx at a time costs one accept per loop iteration, not measured.
 
 - [ ] **Drop the streaming exemption in laravel-spawn.** `TrueAsyncServer::streamContent`
   calls `setNoCompression()` on every `StreamedResponse` as the workaround for #170

include/php_http_server.h

@@ -988,6 +988,12 @@ typedef struct http_server_counters_s {
     uint64_t pause_count_total;
     uint64_t codel_trips_total;
     uint64_t paused_total_ns;
+    /* Connections answered 503 by the hard-cap safety net in the accept
+     * callback. A paused listener accepts nothing, so a nonzero count is a
+     * pause that did not hold. accept_errors_total counts accepts the reactor
+     * reported as failed. */
+    uint64_t accepts_refused_at_cap_total;
+    uint64_t accept_errors_total;
 
     /* The sum and the count are separate fields because the mean is taken after
      * the pool is summed: averaging per-worker averages weights a quiet worker
@@ -1092,6 +1098,8 @@ typedef struct http_server_counters_s {
     X(pause_count_total,                     SUM)            \
     X(codel_trips_total,                     SUM)            \
     X(paused_total_ns,                       SUM)            \
+    X(accepts_refused_at_cap_total,          SUM)            \
+    X(accept_errors_total,                   SUM)            \
     X(tls_handshakes_total,                  SUM)            \
     X(tls_handshake_failures_total,          SUM)            \
     X(tls_handshake_ns_sum,                  SUM)            \

src/http_server_class.c

@@ -1055,11 +1055,16 @@ static void http_server_deadline_tick_stop(http_server_object *server)
     }
 }
 
-/* Backpressure primitives. Calling stop()/start() on a zend_async listen
- * event toggles its presence in the reactor poll set — no fd is closed,
- * so new SYNs accumulate in the kernel backlog while we are paused and
- * are picked up normally when we resume. Hard cap and CoDel share the
- * same pause/resume plumbing; they differ only in their trigger. */
+/* Backpressure primitives. stop() on a zend_async listen event pauses it:
+ * no fd is closed, the reactor stops accepting, and new connections wait in
+ * the kernel backlog until start(). The reactor itself holds the one it had
+ * already taken when the pause began (on Windows an AF_UNIX listener holds
+ * its pending pipe instances, four by default) and delivers it first on
+ * resume. Under SO_REUSEPORT the kernel keeps hashing a paused worker's share
+ * of new connections to its socket: they wait for this worker, or are
+ * dropped once its backlog is full, and other workers do not take them.
+ * Hard cap and CoDel share the same pause/resume plumbing; they differ only
+ * in their trigger. */
 static void http_server_pause_listeners(http_server_object *server, bool drain_connections);
 static void http_server_resume_listeners(http_server_object *server);
 
@@ -1087,7 +1092,8 @@ http_server_aggregate_sample(http_server_object *s,
     }
 }
 
-/* Pauses accept (REUSEPORT hash excludes this worker) and, if the
+/* Pauses accept (see the backpressure note above: under REUSEPORT the
+ * kernel still sends this worker its share) and, if the
  * caller is an overload trigger (drain_connections=true), also bumps
  * the drain epoch so already-accepted connections get a migration
  * signal on their next response. The bool lets future maintenance-
@@ -2219,6 +2225,7 @@ static void http_server_accept_callback(
         /* Accept error — reactor already attached the exception; drop it
          * here to avoid stopping the loop for a transient per-connection
          * failure. */
+        server->counters_live->accept_errors_total++;
         return;
     }
 
@@ -2245,6 +2252,7 @@ static void http_server_accept_callback(
                                "Service Unavailable";
         send(client_fd, response, (int)strlen(response), MSG_NOSIGNAL);
         closesocket(client_fd);
+        server->counters_live->accepts_refused_at_cap_total++;
         /* Make sure listeners are paused so we stop accepting new ones. */
         if (!server->listeners_paused) {
             /* Race fallback — still counts as overload, drain existing. */
@@ -5631,6 +5639,9 @@ ZEND_METHOD(TrueAsync_HttpServer, getTelemetry)
     add_assoc_long (return_value, "codel_trips_total",    (zend_long)server->counters_live->codel_trips_total);
     add_assoc_double(return_value, "paused_total_ms",
                      (double)server->counters_live->paused_total_ns * ns_to_ms);
+    add_assoc_long (return_value, "accepts_refused_at_cap_total",
+                    (zend_long)server->counters_live->accepts_refused_at_cap_total);
+    add_assoc_long (return_value, "accept_errors_total", (zend_long)server->counters_live->accept_errors_total);
     /* This worker's slot — getStats() is what sums them across the pool. */
     const http_server_counters_t *const tc = server->counters_live;
 
@@ -5747,6 +5758,8 @@ ZEND_METHOD(TrueAsync_HttpServer, resetTelemetry)
     server->counters_live->pause_count_total    = 0;
     server->counters_live->codel_trips_total    = 0;
     server->counters_live->paused_total_ns      = 0;
+    server->counters_live->accepts_refused_at_cap_total = 0;
+    server->counters_live->accept_errors_total          = 0;
     server->counters_live->tls_handshakes_total           = 0;
     server->counters_live->tls_handshake_failures_total   = 0;
     server->counters_live->tls_handshake_ns_sum           = 0;

tests/phpt/server/h1/062-h1-paused-listener-leaves-the-backlog.phpt

@@ -0,0 +1,94 @@
+--TEST--
+HTTP/1: a connection that arrives while the listeners are paused waits in the backlog and is served after the resume
+--EXTENSIONS--
+true_async_server
+true_async
+--FILE--
+<?php
+/* setMaxConnections(2) pauses the listeners at the second accept. A third
+ * client that connects during the pause must stay in the kernel backlog: the
+ * server does not accept it, so it gets no 503 from the hard-cap safety net,
+ * and once the two parked handlers answer and their connections close, the
+ * resume accepts it and it gets its 200. */
+
+use TrueAsync\HttpServer;
+use TrueAsync\HttpServerConfig;
+use function Async\spawn;
+use function Async\await;
+use function Async\delay;
+
+require_once __DIR__ . '/../_free_port.inc';
+
+$port = tas_free_port();
+$config = (new HttpServerConfig())
+    ->addListener('127.0.0.1', $port)
+    ->setReadTimeout(10)
+    ->setWriteTimeout(10)
+    ->setKeepAliveTimeout(30)
+    ->setMaxConnections(2)
+    ->setDrainSpreadMs(100)
+    ->setDrainCooldownMs(1000);
+
+$server = new HttpServer($config);
+$server->addHttpHandler(function ($req, $res) {
+    if ($req->getUri() !== '/third') {
+        delay(600);
+    }
+    $res->setStatusCode(200)->setBody('ok');
+});
+
+function read_status($fp): string
+{
+    $buf = '';
+    while (!feof($fp)) {
+        $c = fread($fp, 4096);
+        if ($c === '' || $c === false) {
+            break;
+        }
+        $buf .= $c;
+        if (strpos($buf, "\r\n\r\n") !== false) {
+            break;
+        }
+    }
+    return strtok($buf, "\r\n") ?: '(nothing)';
+}
+
+$client = spawn(function () use ($port, $server) {
+    delay(30);
+
+    $fp1 = stream_socket_client("tcp://127.0.0.1:$port", $e, $es, 3);
+    $fp2 = stream_socket_client("tcp://127.0.0.1:$port", $e, $es, 3);
+    fwrite($fp1, "GET /a HTTP/1.1\r\nHost: x\r\n\r\n");
+    fwrite($fp2, "GET /b HTTP/1.1\r\nHost: x\r\n\r\n");
+    delay(100);
+
+    echo "paused=", (int)$server->getTelemetry()['listeners_paused'], "\n";
+
+    $fp3 = stream_socket_client("tcp://127.0.0.1:$port", $e, $es, 3);
+    fwrite($fp3, "GET /third HTTP/1.1\r\nHost: x\r\n\r\n");
+    stream_set_timeout($fp3, 5);
+
+    foreach ([$fp1, $fp2] as $fp) {
+        stream_set_timeout($fp, 5);
+        echo "first two: ", read_status($fp), "\n";
+    }
+    fclose($fp1);
+    fclose($fp2);
+
+    echo "third: ", read_status($fp3), "\n";
+    fclose($fp3);
+
+    echo "refused at cap: ", $server->getTelemetry()['accepts_refused_at_cap_total'], "\n";
+    $server->stop();
+});
+
+$server->start();
+await($client);
+echo "done\n";
+--EXPECT--
+paused=1
+first two: HTTP/1.1 200 OK
+first two: HTTP/1.1 200 OK
+third: HTTP/1.1 200 OK
+refused at cap: 0
+done