Skip to content

fix(dtls): make reader-thread death visible and fail fast - #38

Merged
QuiteYellow merged 1 commit into
mainfrom
fix/reader-death-visible
Aug 14, 2026
Merged

fix(dtls): make reader-thread death visible and fail fast#38
QuiteYellow merged 1 commit into
mainfrom
fix/reader-death-visible

Conversation

@QuiteYellow

Copy link
Copy Markdown
Owner

Problem

_reader_loop exited silently on any socket error with a bare return. Nothing was logged, and self.conn/self.sock stayed set, so the session still looked open to every caller. Each later get()/post()/ping() passed its conn is None guard, sent its CON, and waited on an Event no thread would ever set — three attempts at four seconds each, then SessionTimeoutError, on every call, forever.

This started biting in v0.1.3. d677c72 moved the session onto connected UDP sockets, and a connected UDP socket surfaces ICMP errors on the next recv. A single ECONNREFUSED from an appliance whose DTLS port is momentarily closed (rebooting, router blip) now kills the reader, where an unconnected socket had simply timed out and looped.

Refs #37 — this is the invisible-failure half of what charettepa is seeing. It does not claim to fix a device-side wedge; it makes the failure visible and fast instead of silent and slow.

Changes

  • Advisory ICMP errnos keep the reader alive. ECONNREFUSED, EHOSTUNREACH, ENETUNREACH, EHOSTDOWN, ENETDOWN log at DEBUG and continue — UDP delivery was never guaranteed and the next datagram usually works.
  • Real reader exits log at WARNING. A close()-driven exit (_stop already set) still returns quietly, so normal teardown stays silent.
  • Callers fail fast. A new _reader_running Event is set in start_reader() and cleared in the loop's finally. get/post/ping/subscribe/refresh_observes now go through _check_live(), which raises SessionClosedError immediately when the reader is gone instead of waiting out the request timeout. Callers that never start a reader (config-flow style) are unaffected — the reader check only applies once _reader_thread is set.

_send_dgram keeps its own conn is None check unchanged: _dispatch_coap calls it to auto-ACK device CONs, so it runs on the reader thread and must not raise during teardown.

Behaviour change for downstream (mbillow/localthings)

A dead reader now raises SessionClosedError, not SessionTimeoutError. SessionClosedError subclasses SessionError/ConnectionError, not TimeoutError, so localthings' _poll_once_defer_reconnect_for three-timeout tolerance is bypassed and it reconnects on the first occurrence rather than after ~2 minutes of a device sitting unavailable. That is the intended outcome, but it changes reconnect timing and downstream should hear it from us.

Tests

New tests/test_dtls_session_reader_death.py drives _reader_loop with a scripted fake socket/conn:

  • advisory ECONNREFUSED survives and the following datagram is still dispatched (one DEBUG line, no WARNING);
  • a fatal EBADF exits with exactly one WARNING and clears _reader_running;
  • get() after reader death raises SessionClosedError in <1s (asserts elapsed time so a regression that reintroduces the wait is caught);
  • close() teardown logs no WARNING;
  • _check_live() with no reader started matches the old conn is None guard.

Full tracked suite: 168 passed.

Not settled here

Whether #37's fridge is wedged by the fixed source port (v0.1.1) or by something the connected socket introduced (v0.1.3). That still needs charettepa's version-pin bisect — but once this ships, the logs will say which of the two the device is actually doing.

The reader loop exited silently on any socket error, leaving conn/sock
set so the session still looked open. Every later get()/post()/ping()
then waited out its full request timeout on a session nobody was
reading, raising SessionTimeoutError on repeat, forever.

This started biting in v0.1.3 (d677c72), which moved to connected UDP
sockets: a connected socket surfaces ICMP errors on recv, so one
ECONNREFUSED from a rebooting appliance now killed the reader.

- Advisory ICMP errnos (ECONNREFUSED/EHOSTUNREACH/...) no longer kill
  the reader; the next datagram usually works.
- Real reader exits log at WARNING; close()-driven exits stay quiet.
- A _reader_running Event lets get/post/ping/subscribe/refresh_observes
  fail fast via _check_live() with SessionClosedError instead of waiting
  out a timeout. Callers that never start a reader are unaffected.

Refs #37
@QuiteYellow

Copy link
Copy Markdown
Owner Author

@mbillow

@QuiteYellow
QuiteYellow merged commit b0d51ab into main Aug 14, 2026
8 checks passed
@mbillow

mbillow commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

mbillow/localthings#368

It’ll get lumped into the next LocalThings release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants