fix(tests): a hardcoded ephemeral port could TCP self-connect, and the syslog startup connect was unbounded (BACKLOG #349, #350; files #351) - #155
Merged
Conversation
…ed port (BACKLOG #349) Both tests asserted a down collector by connecting to a hardcoded high port on the premise "port N is unbound -> connect raises OSError". That premise is unsound on Windows for any port in the dynamic range: SysLogHandler.createSocket issues a BLIND connect (it never bind()s), so the kernel draws the socket's own SOURCE port from that same range. When the allocator hands it the DESTINATION port, TCP simultaneous open connects the socket to itself, connect() succeeds with nothing listening, and `installed` is True. Reproduced on the real syscall path: local == peer, SO_ERROR 0. The contract under test is "an OSError raised while BUILDING the handler is tolerated", not "port N is closed", so the refusal is now injected at the syscall seam. The real _build_syslog_handler, the real tcp/tls/udp dispatch and the real except-OSError warn path are all still exercised; the test is deterministic instead of merely improbable. Patch createSocket, NOT socket.create_connection: SysLogHandler uses getaddrinfo + socket.socket() + sock.connect() and never touches create_connection, so that patch would intercept nothing and leave the flake shipping. The TLS sibling gets the same treatment. It was never safe, only lucky -- after a self-connect it reads back its own ClientHello and dies with ssl.SSLError, an OSError subclass, so its assertions passed by accident.
_TimeoutSysLogHandler captured `timeout=` into self._sock_timeout and then called super().__init__(*args, **kwargs) WITHOUT it, so self.timeout stayed None. In stdlib handlers.py the inet branch runs `if self.timeout: sock.settimeout(self.timeout)` BEFORE sock.connect(sa) -- that is the only thing bounding the STARTUP connect. This class's own settimeout runs in createSocket AFTER super().createSocket() has already returned, so it could bound later sends and reconnects but never the initial connect. _FORWARD_TCP_TIMEOUT = 5.0 exists precisely so a stalled collector cannot block the calling thread (the asyncio event loop), yet the one connect made during engine startup ran under the OS default instead -- contradicting this class's own docstring and _build_syslog_handler's. A collector host that silently DROPS SYNs rather than refusing them would stall engine start. logging.handlers.SysLogHandler.__init__ has accepted `timeout=` all along; it was simply never passed. Routed through kwargs rather than passed explicitly because `timeout` is also SysLogHandler's 4th POSITIONAL parameter, making super().__init__(*args, timeout=...) a possible double-bind that mypy strict rejects. Every construction site here is keyword-only, so this is equivalent at runtime. Found while fixing #349; unrelated to it beyond sharing the module.
…margin tests/test_cluster_failover_sqlserver.py::test_preferred_delay0_wins_expired_lease_race_over_delayed_node sleeps _TTL + 0.15 so the lease is expired by ~0.15s, then requires a node carrying a 0.5s acquire handicap to be rejected. Correctness therefore rests on under 0.35s of wall clock elapsing across a real SQL Server round-trip on a shared CI runner. Filed, deliberately NOT fixed. On the SAME commit the 2022 leg passed while 2025 failed, and a broken delay predicate would fail on both since that logic is backend-version independent -- so the cause is latency versus margin, not the predicate. But that does NOT exonerate the change it fired on: BACKLOG #348 / ADR 0159 adds work at the _acquire chokepoint, the exact connection path this test round-trips through, so it may be the trigger without being wrong. Distinguishing "marginal test tipped by added latency" from "real regression" needs that change's author. Widening the margin now would convert a visible question into a silent one. Same defect class as #349: an environmental assumption asserted as fact.
wshallwshall
enabled auto-merge (squash)
August 2, 2026 19:49
# Conflicts: # docs/BACKLOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three findings from investigating a single CI red, landed as its own PR rather than folded into the PR it fired on — it is a shared-test defect, and coupling them would misattribute it.
BACKLOG #349 — two logging tests could self-connect (fixed)
Both tests asserted "collector is down" by connecting to a hardcoded high port, on the premise "port N is unbound → connect raises OSError." That premise is unsound on Windows for any port in 49152–65535.
SysLogHandler.createSocketissues a blind connect — it never callsbind()— so the kernel draws the socket's own source port from the same dynamic range. When the allocator hands it the destination port, TCP simultaneous open (RFC 793) connects the socket to itself:connect()returns success with nothing listening anywhere, andinstalledisTrue.Reproduced on the real syscall path (
select()writability +SO_ERROR == 0+ non-0.0.0.0local address, avoiding thegetpeername()-on-pending-connect false positive): the socket established withlocal == peer,SO_ERROR 0.The contract under test is "an
OSErrorraised while building the handler is tolerated" — not "port N is closed" — so the refusal now comes from the syscall seam. The real_build_syslog_handler, the real tcp/tls/udp dispatch, and the realexcept OSErrorwarn path are all still exercised.The TLS sibling was fixed too. It was never safe, only lucky: after a self-connect it reads back its own ClientHello and dies with
ssl.SSLError, anOSErrorsubclass, so its assertions passed by accident.The "bind, read the port, close it, reuse it" fix was rejected — it returns a port from the ephemeral range by construction, the exact enabling precondition, and was demonstrated self-connectable on its own output. This repo had already retired that antipattern (
tests/test_load_runner.pydocuments the TOCTOU race verbatim).BACKLOG #350 — the syslog startup connect was unbounded (fixed)
_TimeoutSysLogHandlercapturedtimeout=but never forwarded it tosuper().__init__, soself.timeoutstayedNone. Stdlib runsif self.timeout: sock.settimeout(self.timeout)beforesock.connect(sa)— the only thing bounding the startup connect. The subclass's ownsettimeoutruns aftersuper().createSocket()returns, so it bounds later sends and reconnects but never the initial connect._FORWARD_TCP_TIMEOUT = 5.0existed precisely to stop a stalled collector blocking the event loop; the startup connect ran under the OS default instead, contradicting the class's own docstring.Routed through
kwargsbecausetimeoutis alsoSysLogHandler's 4th positional parameter — the explicit form is a double-bind that mypy strict rejects.BACKLOG #351 — a failover test's 0.35s margin (filed, deliberately not fixed)
Filed only. On the same commit the SQL Server 2022 leg passed while 2025 failed, so the cause is latency vs margin rather than the delay predicate. But that does not exonerate the change it fired on (ADR 0159 adds work at the exact
_acquirepath the test round-trips through), and distinguishing "marginal test tipped by added latency" from "real regression" needs that change's author. Widening the margin now would turn a visible question into a silent one.Scope correction
The red was previously labelled repo-wide. It is not: one branch, one run, one job — ever, across two exhaustive scans with verified positive controls. Genuinely repo-wide reds here show 129 / 69 / 27 occurrences. It also never blocked the PR it appeared on —
git diff origin/main <that head> -- tests/test_logging.py messagefoundry/logging_setup.pyis empty. Re-running the one job was the entire remedy.The mechanism is confirmed sufficient, not confirmed observed — the failing job carries zero port telemetry, and a transient listener from any runner process yields an identical observable. Not chased further, because the fix is identical under both surviving hypotheses.
Verification
ruff format --check/ruff check— cleanmypystrict — no errors in changed files; total at the pre-existing 21-error baseline (pynetdicomstubs)pytest— 119 passing across every module importinglogging_setupcreateSocketpatched to succeed,installed=True(reproducing the CI failure); patched to raise,installed=False— the assertion discriminatesbacklog_status_check.pyfail naming #349, so its green means it can see these items🤖 Generated with Claude Code