Skip to content

Add browser-side audio response timing diagnostics - #728

Open
ChenYingHan1121 wants to merge 3 commits into
sysprog21:mainfrom
ChenYingHan1121:feat/browser-audio-timing-712
Open

ChenYingHan1121 wants to merge 3 commits into
sysprog21:mainfrom
ChenYingHan1121:feat/browser-audio-timing-712

Conversation

@ChenYingHan1121

@ChenYingHan1121 ChenYingHan1121 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add opt-in browser-side audio response timing diagnostics to help developers investigate the delay between the interviewer's speaking state and the first detectable remote audio activity.

The diagnostic is disabled by default and does not change the candidate-facing interface or avatar playback behavior.

Changes

  • Add web/audio-response-timing.js to measure the browser-observed delay using Web Audio analysis.
  • Enable the diagnostic only when ?audioTiming=1 is present in the URL.
  • Start timing when LiveKit reports lk.agent.state=speaking and report the elapsed time when audio activity crosses the configured detection threshold.
  • Cancel incomplete measurements when audio monitoring is interrupted or the connection becomes unavailable.
  • Handle audio track changes, reconnections, and browser audio context lifecycle events.
  • Log timing results to the developer console without modifying the interview UI.

Validation

  • 65 relevant browser tests passed.
  • ESLint and formatting checks passed.
  • Git pre-commit and pre-push checks passed.
  • Local interactive review completed.

Limitations

  • Timing is browser-observed and does not represent end-to-end model or server latency.
  • Actual LiveKit audio timing has not yet been verified in a live interview.
  • A review identified a remaining P2 test gap: reconnect coverage should exercise the registered room callbacks directly.

Close #712


Summary by cubic

Adds an opt-in, browser-side diagnostic that measures the delay between the interviewer's speaking state and the first detectable remote audio activity, helping developers investigate response timing without changing the candidate-facing interface or avatar playback. It is disabled by default and only activates when the URL contains ?audioTiming=1.

  • Starts timing when LiveKit reports lk.agent.state=speaking and only reports after a quiet frame is followed by audio above a configured threshold, so pre-existing audio never inflates the gap.
  • Cancels incomplete measurements on interruption, page hide, connection drops, and track changes, and ignores stale room events that could release a newer monitor.
  • Reuses the shared startMediaMeter loop, which now supports configurable sampling, a deferred first frame, and explicit cleanup.
  • Logs results to the developer console; timing is browser-observed and not end-to-end model or server latency.

Recovery coverage

  • Added integration tests confirming that monitor setup failures and interviewer unavailability clear stale timing state and allow a later speaking turn to be measured.

Written for commit 3094341. Summary will update on new commits.

View guided diff

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

jserv

This comment was marked as outdated.

jserv

This comment was marked as resolved.

jserv

This comment was marked as resolved.

Comment thread web/interview.js
});
}
attachAvatarAnalyser(track, participant);
attachAudioTiming(track);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds a second AudioContext, a second requestAnimationFrame loop and a second pair of gesture listeners on the same mediaStreamTrack that attachAvatarAnalyser (via bindAnalyser in web/avatar/stage.js) already analyses with the same peakLevel over a time-domain buffer. Keeping it separate is defensible because the avatar analyser is absent when the avatar is hidden or retired, but as a follow-up consider sharing one context (or one per-frame peak) between the two, so the suspended-context and gesture-resume handling lives in one place instead of drifting apart.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. I agree that the audio timing diagnostic currently uses a separate AudioContext and animation-frame loop from the avatar analyser.

I've kept them independent in this revision because the timing diagnostic is opt-in and should remain available even when the avatar is hidden or retired.

I also refactored the timing diagnostic to reuse the shared startMediaMeter() implementation in web/mic-meter.js, reducing duplicated sampling and lifecycle logic.

Sharing an AudioContext or per-frame peak value with the avatar analyser would be a useful follow-up optimization, particularly to consolidate suspended-context and gesture-resume handling. I've left the avatar analyser unchanged in this PR to avoid expanding the scope or affecting existing lip-sync behavior.

jserv

This comment was marked as resolved.

Measure the browser-observed delay between the interviewer's
speaking state and detectable remote audio activity.

Keep the diagnostic opt-in and independent of avatar playback
so developers can investigate response timing without changing
the candidate interface.

Close sysprog21#712

Change-Id: Iac1dc45850391ef264036f503c263cfd97a58bcf
Correct timing results when remote audio is already active or
unavailable, and ensure stale room events cannot release a newer
monitor.

Reuse the existing media meter to avoid duplicate audio analysis
logic while preserving independent remote audio monitoring.

Add regression tests for reconnection, cancellation, silent audio,
and track lifecycle behavior.

Change-Id: I20eb8e10099dc0bcb584d81431e71726a51223de
@ChenYingHan1121
ChenYingHan1121 force-pushed the feat/browser-audio-timing-712 branch from ed71213 to b93c715 Compare November 16, 2026 15:36
Comment thread web/interview.js
);
} catch (error) {
releaseAudioTiming();
console.warn("[audio-timing] Monitor unavailable", error);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test reaches this catch: with releaseAudioTiming() removed here, every browser test still passes. Without it, a failed monitor setup leaves audioTimingTrack pointing at the track, so the Reconnected handler's attachAudioTiming(jimTrack) for that same track returns early on audioTimingTrack === track and the diagnostic stays off for the rest of the session. A case in audio-response-timing-integration.test.js whose startAudioActivityMonitor throws once, asserting that currentTrack() is null and that a later Reconnected with the track still live builds a monitor, would pin it.

Comment thread web/interview.js
return;
}
if (!agent) {
audioTiming?.reset("interviewer unavailable");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reset is not reached by any test either: replacing it with nothing leaves every test that drives updateAgentState passing. It is what clears speaking inside the timer when Jim leaves mid-reply; without it, an interviewer who comes back already publishing speaking hits the nextSpeaking === speaking early return in speakingChanged, so that turn is neither measured nor explained in the console. An integration case where roomInterviewer returns null during a pending turn, asserting the interviewer unavailable diagnostic and that the next speaking transition is measured, would cover it.

@jserv

jserv commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Some review comments on this pull request are not settled yet. Please refine the change per the review comments listed below, reply on each thread with what you changed or why you did not, and mark each thread resolved yourself once it is settled (Resolve conversation, at the bottom of the thread). The pull request stays blocked while review threads are open.

Not addressed at the current head:

Addressed at the current head; reply with what changed and resolve:

Checked at b93c715.

Add integration tests for monitor setup failures and interviewer
unavailability. Verify that both paths clear stale timing state
and allow a later speaking turn to be measured.

These tests cover the recovery paths identified during PR sysprog21#728
review and guard against regressions in cleanup and reset logic.

Change-Id: Ie603ddde2af7aa5c4650733f6df71464d0b08a8e

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/browser/audio-response-timing-integration.test.js">

<violation number="1" location="tests/browser/audio-response-timing-integration.test.js:472">
P2: This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit `ParticipantDisconnected` and await its microtask instead.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Re-trigger cubic

monitor.options.onAvailable();
assert.equal(f.api.ready(), true);
f.agent.attributes["lk.agent.state"] = "speaking";
f.room.emit("ParticipantAttributesChanged");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit ParticipantDisconnected and await its microtask instead.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/browser/audio-response-timing-integration.test.js, line 472:

<comment>This test bypasses the real interviewer-disconnect recovery path, so regressions in its deferred cleanup or reconnect guard can still pass. Emit `ParticipantDisconnected` and await its microtask instead.</comment>

<file context>
@@ -436,3 +448,62 @@ test("callbacks from a replaced track cannot cancel or measure the current track
+  monitor.options.onAvailable();
+  assert.equal(f.api.ready(), true);
+  f.agent.attributes["lk.agent.state"] = "speaking";
+  f.room.emit("ParticipantAttributesChanged");
+  monitor.options.onQuiet();
+  f.advance(80);
</file context>
Suggested change
f.room.emit("ParticipantAttributesChanged");
f.room.emit("ParticipantDisconnected", f.agent);
await Promise.resolve();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add browser-side audio response timing diagnostics

2 participants