Skip to content

End capture when the speaker stops, and think until speech actually starts#216

Merged
RobVanProd merged 1 commit into
mainfrom
feat/responsive-capture-and-thinking-face
Jul 26, 2026
Merged

End capture when the speaker stops, and think until speech actually starts#216
RobVanProd merged 1 commit into
mainfrom
feat/responsive-capture-and-thinking-face

Conversation

@RobVanProd

Copy link
Copy Markdown
Owner

Two things reported from live use: replies cut off at 4.8 s, and no thinking face while a reply is being prepared.

Capture length

Three things stacked up:

  1. VoiceActivityEndpointConfig::maximumCaptureMs was 4800, so any sentence past ~4 s was truncated mid-word regardless of whether the speaker had finished. Now 12000 — a backstop, not the normal path. Capture still ends on 550 ms of trailing silence as soon as you stop.
  2. kWakeMwwDedicatedCaptureChunks was 96 (9.6 s). Now 130 (13 s), so the chunk ceiling sits behind the endpoint rather than in front of it.
  3. The endpoint only ran for conversation replies, not the initial wake-gated utterance. So the first capture after a wake phrase always ran its full fixed length — a two-second question still cost the whole window before anything reached the bridge. Both paths now endpoint on silence.

Measured, capture length vs how long the speaker actually talks:

spoke before after
1.0 s 1.6 s 1.6 s
4.0 s 4.6 s 4.6 s
6.0 s 4.8 s 6.6 s
8.0 s 4.8 s 8.6 s
11.0 s 4.8 s 11.6 s

Before, anything past ~4 s ended with reason=max_duration and lost the remainder — 11 s of speech lost 6.2 s. After, every case ends with trailing_silence and a constant 0.6 s tail.

Byte budget: 12 s of 16 kHz mono PCM is 384 KB, inside the 512 KB limit on both firmware (kBridgeAudioUplinkMaxBytes) and bridge (DEFAULT_MAX_AUDIO_BYTES).

Thinking face

response_start only means the reply text is ready — TTS still has to render. That gap was spent in Speak, with a speaking mouth and no sound.

He now holds the thinking face through it and moves to Speak on the first audio of the response, whether that arrives as an audio_stream_start or a mouth audio frame. Think already reads clearly: eyes up and to the side (FaceAnimator Think saccade targets), amber body light with a chase pulse.

Adds IntentEngine::setMode for state changes that continue something already accounted for, so the emotional event isn't applied twice. It refreshes the event clock so the mode-decay timers don't unwind it, and it deliberately cannot wake a sleeping robot — waking stays event-driven.

The awaiting-first-speech flag is cleared on response_end, error, and the downlink abort path, so it can't leak into the next turn. If audio never arrives at all, the 90 s conversation-stall recovery from #214 still applies.

Verification

  • 2 new VAD tests: capture tracks speech length and always ends on trailing silence; default ceiling is generous but inside the byte limit
  • 1 new IntentEngine test: setMode moves without an event and is ignored while asleep
  • All 10 sleep/mode tests and 10 IntentEngine tests pass
  • Builds for ESP32-S3: RAM 56.8%, Flash 43.2%

Existing VAD tests all set explicit configs, so the default change doesn't affect them.

Note

The main.cpp changes (endpoint gating, thinking-face deferral) are only exercised on target — they sit in the bridge output handler, which isn't in the native test build. The endpoint timing itself is covered natively.

🤖 Generated with Claude Code

…tarts

Two things reported from live use: replies were cut off at 4.8 s, and the face
did not look like it was thinking while a reply was being prepared.

Capture length:
- The voice-activity endpoint's maximumCaptureMs was 4800, so any sentence past
  roughly four seconds was truncated mid-word regardless of whether the speaker
  had finished. It is now 12000, which is a backstop rather than the normal
  path: capture still ends on 550 ms of trailing silence as soon as the speaker
  stops. 12 s of 16 kHz mono PCM is 384 KB, inside the 512 KB uplink limit.
- The dedicated capture chunk ceiling moves from 96 (9.6 s) to 130 (13 s) so it
  sits behind the endpoint rather than in front of it.
- The endpoint now runs for the initial wake-gated utterance too, not only for
  conversation replies. Previously the first capture after a wake phrase always
  ran its full fixed length, so a two-second question still cost the whole
  window before anything reached the bridge.

Measured, capture length against how long the speaker actually talks:

  spoke  1.0s   before 1.6s    after 1.6s
  spoke  4.0s   before 4.6s    after 4.6s
  spoke  6.0s   before 4.8s    after 6.6s
  spoke  8.0s   before 4.8s    after 8.6s
  spoke 11.0s   before 4.8s    after 11.6s

Before, anything longer than about four seconds ended with reason max_duration
and lost the remainder. After, every case ends with trailing_silence and a
constant 0.6 s tail.

Thinking face:
- response_start only means the reply text is ready; TTS still has to render.
  That gap was spent in Speak, with a speaking mouth and no sound. He now holds
  the thinking face through it and moves to Speak on the first audio of the
  response, whether that arrives as an audio stream or a mouth frame.
- Adds IntentEngine::setMode for state changes that are a continuation of
  something already accounted for, so the emotional event is not applied twice.
  It refreshes the event clock so the mode-decay timers do not unwind it, and it
  deliberately cannot wake a sleeping robot; waking stays event-driven.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@RobVanProd
RobVanProd merged commit 6d39af7 into main Jul 26, 2026
8 checks passed
@RobVanProd
RobVanProd deleted the feat/responsive-capture-and-thinking-face branch July 26, 2026 06:49
@RobVanProd

Copy link
Copy Markdown
Owner Author

Post-merge hardware/bridge handoff from PR #199: the longer endpoint is present and active (12,000 ms maximum, 550 ms trailing silence, 13-second dedicated capture ceiling). A boundary race remains because BridgeWakeGate also expires at 12,000 ms. At the exact ceiling it can close audio uplink before the final capture chunk submit. Two live captures reached the host with their exact declared totals (118 + 113 = 231 chunks; 369,600 bytes), but robot counters increased by 80 bridge_uplink_errors with bridge_uplink_last_error=audio_uplink_not_active and by 2 mww_uplink_submit_failed; queue failures stayed zero. This points to local firmware gate ordering, not Cloudflare/host transport loss. PR #199 keeps firmware byte-for-byte aligned with main and will not patch it; zero new uplink errors remains a physical-promotion gate for the firmware owner.

@RobVanProd

Copy link
Copy Markdown
Owner Author

Qualification provenance follow-up: PR #199 now refuses the historical pre-216 ce66f8a / 69d3... acceptance pair. Before the supervised bridge session, firmware ownership needs to record the exact private paired firmware.bin SHA-256 together with source commit 6d39af7 (or a later main descendant) in docs/FIRST_DEPLOY_STATUS.md, after closing the 12-second gate race. The live robot currently reports the old configured ota_expected_sha256 even though the longer capture behavior is present, so that field alone is not sufficient source-to-binary provenance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant