Repository navigation
Add browser-side audio response timing diagnostics - #728
ChenYingHan1121 wants to merge 3 commits into
Conversation
| }); | ||
| } | ||
| attachAvatarAnalyser(track, participant); | ||
| attachAudioTiming(track); |
There was a problem hiding this comment.
This builds a second AudioContext, a second requestAnimationFrame loop and a second pair of gesture listeners on the same mediaStreamTrack that attachAvatarAnalyser (via bindAnalyser in web/avatar/stage.js) already analyses with the same peakLevel over a time-domain buffer. Keeping it separate is defensible because the avatar analyser is absent when the avatar is hidden or retired, but as a follow-up consider sharing one context (or one per-frame peak) between the two, so the suspended-context and gesture-resume handling lives in one place instead of drifting apart.
There was a problem hiding this comment.
Thanks for pointing this out. I agree that the audio timing diagnostic currently uses a separate AudioContext and animation-frame loop from the avatar analyser.
I've kept them independent in this revision because the timing diagnostic is opt-in and should remain available even when the avatar is hidden or retired.
I also refactored the timing diagnostic to reuse the shared startMediaMeter() implementation in web/mic-meter.js, reducing duplicated sampling and lifecycle logic.
Sharing an AudioContext or per-frame peak value with the avatar analyser would be a useful follow-up optimization, particularly to consolidate suspended-context and gesture-resume handling. I've left the avatar analyser unchanged in this PR to avoid expanding the scope or affecting existing lip-sync behavior.
Measure the browser-observed delay between the interviewer's speaking state and detectable remote audio activity. Keep the diagnostic opt-in and independent of avatar playback so developers can investigate response timing without changing the candidate interface. Close sysprog21#712 Change-Id: Iac1dc45850391ef264036f503c263cfd97a58bcf
Correct timing results when remote audio is already active or unavailable, and ensure stale room events cannot release a newer monitor. Reuse the existing media meter to avoid duplicate audio analysis logic while preserving independent remote audio monitoring. Add regression tests for reconnection, cancellation, silent audio, and track lifecycle behavior. Change-Id: I20eb8e10099dc0bcb584d81431e71726a51223de
ed71213 to
b93c715
Compare
| ); | ||
| } catch (error) { | ||
| releaseAudioTiming(); | ||
| console.warn("[audio-timing] Monitor unavailable", error); |
There was a problem hiding this comment.
No test reaches this catch: with releaseAudioTiming() removed here, every browser test still passes. Without it, a failed monitor setup leaves audioTimingTrack pointing at the track, so the Reconnected handler's attachAudioTiming(jimTrack) for that same track returns early on audioTimingTrack === track and the diagnostic stays off for the rest of the session. A case in audio-response-timing-integration.test.js whose startAudioActivityMonitor throws once, asserting that currentTrack() is null and that a later Reconnected with the track still live builds a monitor, would pin it.
| return; | ||
| } | ||
| if (!agent) { | ||
| audioTiming?.reset("interviewer unavailable"); |
There was a problem hiding this comment.
This reset is not reached by any test either: replacing it with nothing leaves every test that drives updateAgentState passing. It is what clears speaking inside the timer when Jim leaves mid-reply; without it, an interviewer who comes back already publishing speaking hits the nextSpeaking === speaking early return in speakingChanged, so that turn is neither measured nor explained in the console. An integration case where roomInterviewer returns null during a pending turn, asserting the interviewer unavailable diagnostic and that the next speaking transition is measured, would cover it.
|
Some review comments on this pull request are not settled yet. Please refine the change per the review comments listed below, reply on each thread with what you changed or why you did not, and mark each thread resolved yourself once it is settled (Resolve conversation, at the bottom of the thread). The pull request stays blocked while review threads are open. Not addressed at the current head:
Addressed at the current head; reply with what changed and resolve:
Checked at b93c715. |
Add integration tests for monitor setup failures and interviewer unavailability. Verify that both paths clear stale timing state and allow a later speaking turn to be measured. These tests cover the recovery paths identified during PR sysprog21#728 review and guard against regressions in cleanup and reset logic. Change-Id: Ie603ddde2af7aa5c4650733f6df71464d0b08a8e
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/browser/audio-response-timing-integration.test.js">
<violation number="1" location="tests/browser/audio-response-timing-integration.test.js:472">
P2: This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit `ParticipantDisconnected` and await its microtask instead.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
View guided diff | Re-trigger cubic
| monitor.options.onAvailable(); | ||
| assert.equal(f.api.ready(), true); | ||
| f.agent.attributes["lk.agent.state"] = "speaking"; | ||
| f.room.emit("ParticipantAttributesChanged"); |
There was a problem hiding this comment.
P2: This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit ParticipantDisconnected and await its microtask instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/browser/audio-response-timing-integration.test.js, line 472:
<comment>This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit `ParticipantDisconnected` and await its microtask instead.</comment>
<file context>
@@ -436,3 +448,62 @@ test("callbacks from a replaced track cannot cancel or measure the current track
+ monitor.options.onAvailable();
+ assert.equal(f.api.ready(), true);
+ f.agent.attributes["lk.agent.state"] = "speaking";
+ f.room.emit("ParticipantAttributesChanged");
+ monitor.options.onQuiet();
+ f.advance(80);
</file context>
| f.room.emit("ParticipantAttributesChanged"); | |
| f.room.emit("ParticipantDisconnected", f.agent); | |
| await Promise.resolve(); |
Summary
Add opt-in browser-side audio response timing diagnostics to help developers investigate the delay between the interviewer's speaking state and the first detectable remote audio activity.
The diagnostic is disabled by default and does not change the candidate-facing interface or avatar playback behavior.
Changes
web/audio-response-timing.jsto measure the browser-observed delay using Web Audio analysis.?audioTiming=1is present in the URL.lk.agent.state=speakingand report the elapsed time when audio activity crosses the configured detection threshold.Validation
Limitations
Close #712
Summary by cubic
Adds an opt-in, browser-side diagnostic that measures the delay between the interviewer's speaking state and the first detectable remote audio activity, helping developers investigate response timing without changing the candidate-facing interface or avatar playback. It is disabled by default and only activates when the URL contains
?audioTiming=1.lk.agent.state=speakingand only reports after a quiet frame is followed by audio above a configured threshold, so pre-existing audio never inflates the gap.startMediaMeterloop, which now supports configurable sampling, a deferred first frame, and explicit cleanup.Recovery coverage
Written for commit 3094341. Summary will update on new commits.