Skip to content

wake: keep the spotter stream alive on a busy host - #26

Merged
yawaflua merged 2 commits into
yawaflua:stagefrom
ryysel:pr/wake-keepalive
Nov 5, 2026
Merged

yawaflua merged 2 commits into
yawaflua:stagefrom
ryysel:pr/wake-keepalive

Conversation

@ryysel

@ryysel ryysel commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Each connection now feeds audio through a dedicated single-thread executor (feed_all), so Vosk decoding and the Whisper frame math never run on the asyncio event loop.
  • serve() gets an explicit ping_interval=20, ping_timeout=60 (was the library default of 20/20).
  • The client's audio Write timeout goes from one second to five: the websocket library closes the whole connection when a write deadline expires, so a single slow write used to kill the spotter stream.

Why

On a busy host the server logs looked like this every few minutes:

wake word spotter stream ended err="received close frame: status = StatusInternalError and reason = \"keepalive ping timeout\""

While the loop is inside KaldiRecognizer.AcceptWaveform, it cannot process the pong for the spotter's keepalive ping. The websockets server then closes the stream, the agent reconnects, and any wake word spoken during that window is lost. The box reached load ~35 on 8 cores during a RAID check and a concurrent build, so this was reproducible.

Moving the decode to a worker keeps frame order (one worker per connection) and leaves the event loop free to answer pings.

The second half is the other side of the same stall: under load the spotter may stop reading for a moment, the client's one-second write deadline expires, and the library tears the connection down. Five seconds rides out a spike; a truly dead peer still fails fast enough for runWake to reconnect.

Tests

  • python3 -m py_compile deploy/wake/wake_server.py, go build ./internal/wake/.
  • Live check on the deployment: a synthetic «Маруся» streamed to ws://…:2700 returns {"keyword": "маруся", "engine": "vosk", "score": 1.0}; the same stream with «Алиса» and keywords [маруся] returns no detection.
  • With both halves deployed, the spotter connection stayed up through a six-core CPU load test; before, it dropped within minutes under load.

No protocol change.

@ryysel
ryysel force-pushed the pr/wake-keepalive branch from 8d49fe0 to 398ca40 Compare November 1, 2026 20:36
@yawaflua
yawaflua merged commit 279ebca into yawaflua:stage Nov 5, 2026
7 checks passed
@yawaflua yawaflua mentioned this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants