Harden HTTP server against up-but-not-serving failure modes

[johnathan-arsenault]

Problem

The agent process could stay alive while no longer accepting connections on its HTTP port. Investigation of the REST sink server lifecycle (Agent::start → RestService::start → Server::start/listen/accept → AsyncContext worker loop) turned up three independent causes.

Root causes & fixes

#	Root cause	Fix
1	Accept loop re-armed async_accept only on the success path, so any accept error (fd exhaustion/EMFILE, ECONNABORTED) permanently stopped the listener while the process stayed alive. Most likely cause of intermittent "up but not serving" outages.	Recoverable errors now re-arm after a 100ms backoff (avoids hot-looping while a transient condition clears); operation_aborted (shutdown) still stops cleanly.
2	Bind failure used non-throwing asio overloads and returned silently — start()'s try/catch never fired, so a failed bind (EADDRINUSE, bad ServerIp, privileged port) left the process running but never listening.	listen() now throws on open/bind/listen/set_option failure → start() logs fatal and std::exit(1) for supervisor restart.
3	io_context worker: a throwing completion handler unwound the only worker thread (default WorkerThreads=1), silencing all async serving while the main thread idled.	Worker now catches exceptions and resumes run() (valid per asio: run() may be re-called after a handler throws).

Plus: Server::fail() now logs the failing-step description (previously discarded, making bind/accept failures undiagnosable), and a new getAcceptRearmCount() exposes accept-error recoveries for diagnostics.

Tests

Genuine regression tests added for each fix:

• HttpServerTest.accept_loop_rearms_after_recoverable_error — asserts the re-arm counter goes 0→1 after an injected recoverable accept error. Verified it fails when Fix Jarsenault debify #1 is reverted (counter stays 0) and passes when restored.
• HttpServerTest.listen_throws_when_port_is_already_bound — asserts listen() throws on a bind conflict.
• AsyncContextTest.worker_survives_a_thrown_handler (new async_context_test.cpp) — asserts a second handler still runs after a prior handler throws.

Validation

Built and tested via the Ubuntu-22.04 CI-parity Docker image (gcc-13, -static-libgcc -static-libstdc++ — matches edge-device link requirements): 756/756 tests pass, exit 0.

Notes for reviewers

• Behavior change in Fix pull in changes from parent fork #2: the agent now exits on an unbindable port instead of running headless. This is correct under systemd/a supervisor. If unsupervised operation where the agent should stay up on a bind failure is desired, this should be gated behind a config option — flagging for review.

🤖 Generated with Claude Code