diff --git a/CLAUDE.md b/CLAUDE.md index 7f79d73..a2c2c36 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -60,16 +60,24 @@ Handler registration lives in [src/main/ipc/](src/main/ipc/) - one file per doma ### Transcription and Suggestion Flow -[src/main/services/transcript.service.ts](src/main/services/transcript.service.ts) is the central orchestrator. `transcriptService.ingest(channel, type, text)` merges both audio channels, deduplicates overlapping segments, and triggers `liveSuggestionService.startGenerateSuggestion()` when a final `Other` transcript arrives - with a `LIVE_SUGGESTION_GAP_MS` guard that suppresses the call if Self spoke recently. +[src/main/services/transcript.service.ts](src/main/services/transcript.service.ts) is the central orchestrator. `transcriptService.ingest(channel, type, text)` merges both audio channels, deduplicates overlapping segments, and decides whether a final `Other` transcript is worth answering - with a `LIVE_SUGGESTION_GAP_MS` guard that suppresses the call if Self spoke recently. - `ch_0` = `Speaker.Other` (interviewer, captured via loopback audio) - `ch_1` = `Speaker.Self` (candidate, captured via microphone) +**Not every interviewer turn needs an answer**, and deciding that is a cascade, cheapest stage first. `classifyInterviewerTurn()` ([src/main/utils/interviewer-turn.ts](src/main/utils/interviewer-turn.ts)) runs in-process on the *merged* turn and returns one of three verdicts: `Skip` drops the turn outright with no request and no card, `Answer` generates immediately, and `Uncertain` parks on an `INTERVIEWER_TURN_SETTLE_MS` timer that any further `ch_0` final re-arms. The `NO_SUGGESTION_NEEDED` sentinel is the *last* stage of the same cascade, not the only one. + +Three things this ordering buys, all of which the sentinel alone could not. A turn caught at `Skip` costs no upload of the profile and context, no model call, and never reaches the panel, so nothing flashes on screen and is retracted. A question the ASR split across two finals - an ASR final is an acoustic endpoint, not the end of a thought - is classified whole instead of firing a request on the fragment that the continuation immediately aborts. And a completed question skips the settle wait entirely, so the latency is paid only by turns that are genuinely ambiguous. + +The classifier is deliberately asymmetric, and `test/interviewer-turn.test.mjs` pins both halves. A filler that slips through costs one request and a card that flashes; a question misread as filler produces *nothing at all*, mid-interview, with no error anywhere. So `Skip` is returned only when the backchannel lexicon consumes the whole turn from the front, and everything it cannot fully consume falls through rather than being guessed at. + +`Answer` and `Uncertain` both reach the backend, as `turn_verdict` on `GenerateLiveSuggestionRequest` ([types/llm.ts](src/main/types/llm.ts) mirrors the wire values `answer` / `uncertain`; `Skip` never becomes a request and has no wire value). `Answer` tells the backend to trust the client and skip its own classifier; `Uncertain` asks it to run one. The backend's decision is speculative - it runs *beside* the generation it might cancel, not in front of it - and the client cooperates by holding the card back: `generateSuggestion()` in [suggestion-live.service.ts](src/main/services/suggestion-live.service.ts) does not append a `Pending` card on request start. It arms a `LIVE_SUGGESTION_RENDER_DELAY_MS` timer instead, so a turn the backend suppresses within that window produces no card at all rather than one that flashes and is retracted - the exact failure this whole cascade exists to remove. Any real write (loading state once headers arrive, a streamed chunk, an error) cancels the timer and renders immediately through `publish()`; a `Stopped` state from being superseded before ever rendering goes through `refresh()` instead, which is a no-op unless a card already exists, so a card the candidate never saw pending does not appear only to say it was cancelled. + Action suggestions are independent of transcripts - triggered by screenshot captures (up to `ACTION_SUGGESTION_MAX_CAPTURES` = 4 images per request). **Professional mode** (`professionalMode` in ConfigStore, off by default) asks the backend for hints - a headline plus keyword bullets - instead of full sentences. Both suggestion services read the flag once at the top of `generateSuggestion` and send it as `mode` on the request; the backend defaults it to `normal`, so the field is safe to omit against an older deployment. -The `NO_SUGGESTION_NEEDED` sentinel goes through `isNoSuggestionSentinel()` ([src/main/utils/suggestion-sentinel.ts](src/main/utils/suggestion-sentinel.ts)) rather than a direct comparison. It is prefix-matched because it runs on every streamed chunk, and it strips leading markdown first: the professional prompt asks for a bold headline on line 1, so a model that carries that format over emits `**NO_SUGGESTION_NEEDED**` and a bare match would leave the sentinel on screen as a card. `test/suggestion-sentinel.test.mjs` pins both halves - the wrapped forms are suppressed, real answers are not. +The `NO_SUGGESTION_NEEDED` sentinel goes through `isNoSuggestionSentinel()` ([src/main/utils/suggestion-sentinel.ts](src/main/utils/suggestion-sentinel.ts)) rather than a direct comparison. It backs up the deterministic gate above for turns the lexicon cannot settle, and it is in-band by nature - a control decision travelling in the answer stream - which is why it is the fallback rather than the mechanism. It is prefix-matched because it runs on every streamed chunk, and it strips leading markdown first: the professional prompt asks for a bold headline on line 1, so a model that carries that format over emits `**NO_SUGGESTION_NEEDED**` and a bare match would leave the sentinel on screen as a card. `test/suggestion-sentinel.test.mjs` pins both halves - the wrapped forms are suppressed, real answers are not. Both live modes render through `SafeMarkdown`, the same component the action panel uses. The normal-mode prompt asks for plain text *with light formatting*, so any bold or bullet the model reached for used to land on screen as literal asterisks. Prose is passed through `withHardBreaks()` ([src/renderer/lib/suggestions.ts](src/renderer/lib/suggestions.ts)) first: Markdown folds a single newline into a space, and the `whitespace-pre-wrap` rendering it replaced showed every newline the model emitted. diff --git a/src/main/consts.ts b/src/main/consts.ts index de84c7c..82217b0 100644 --- a/src/main/consts.ts +++ b/src/main/consts.ts @@ -41,9 +41,30 @@ export const SELF_PARTIAL_STALE_MS = 15_000; // export read the full transcript from app state. export const TRANSCRIPT_UPLOAD_LIMIT = 60; +// How long an interviewer turn that is neither a clear question nor clear filler is allowed to +// settle before it is answered. An ASR final is an acoustic endpoint, not the end of a thought, so +// a question broken by a thinking pause ("So tell me about" ... "your Kafka work") arrives as two +// finals: without this the first fires a request that the second immediately aborts, and the card +// rewrites itself under the candidate mid-read. +// +// Paid for only by turns that reach TurnVerdict.Uncertain. A turn ending in a question mark, or a +// punctuated directive, is answered with no added latency at all - see interviewer-turn.ts. +export const INTERVIEWER_TURN_SETTLE_MS = 700; + // Suggestion constants export const LIVE_SUGGESTION_GAP_MS = 2000; export const LIVE_SUGGESTION_NO_SUGGESTION = 'NO_SUGGESTION_NEEDED'; + +// How long a live-suggestion card waits before it is allowed to render at all. The backend's +// speculative gate (LLMService.turn_needs_answer) runs beside the answer generation and can +// suppress the whole response after it has already started - within roughly this long. Rendering +// a Pending card immediately would make every suppressed turn flash a spinner and then vanish, +// which is worse than the delay: a card that disappears reads as broken, one that never appears +// reads as nothing having happened, which is correct for a "yeah, got it". +// +// Not a cap on generation - a genuinely slow real answer still renders once this fires, it just +// starts one step behind instead of at the first byte. +export const LIVE_SUGGESTION_RENDER_DELAY_MS = 900; export const ACTION_SUGGESTION_MAX_CAPTURES = 4; export const ACTION_TIMEOUT_MS = 30_000; // 30 seconds diff --git a/src/main/services/suggestion-live.service.ts b/src/main/services/suggestion-live.service.ts index bfebade..d7a0b6d 100644 --- a/src/main/services/suggestion-live.service.ts +++ b/src/main/services/suggestion-live.service.ts @@ -1,12 +1,17 @@ import { LLMApi } from '../api/llm.js'; import { + LIVE_SUGGESTION_RENDER_DELAY_MS, LIVE_SUGGESTION_TTFB_MS, SUGGESTION_STALL_MS, TRANSCRIPT_UPLOAD_LIMIT, } from '../consts.js'; import { configStore } from '../store/config.store.js'; import { LiveSuggestion, Speaker, SuggestionState, Transcript } from '../types/app-state.js'; -import { GenerateLiveSuggestionRequest, SuggestionMode } from '../types/llm.js'; +import { + GenerateLiveSuggestionRequest, + RequestTurnVerdict, + SuggestionMode, +} from '../types/llm.js'; import { DateTimeUtil } from '../utils/datetime.js'; import { getSuggestionErrorMessage } from '../utils/suggestion-error.js'; import { isNoSuggestionSentinel } from '../utils/suggestion-sentinel.js'; @@ -49,7 +54,8 @@ class LiveSuggestionService { private async generateSuggestion( taskId: string, controller: AbortController, - transcripts: Transcript[] + transcripts: Transcript[], + turnVerdict: RequestTurnVerdict ): Promise { // No empty-transcript guard here on purpose. startGenerateSuggestion already returns // before registering a task, and a second check would return ahead of the finally that @@ -71,8 +77,33 @@ class LiveSuggestionService { mode, }; - // Append initial suggestion - this.appendSuggestion(timestamp, suggestion, epoch); + // The card is *not* appended yet. A turn the backend suppresses resolves in a few hundred + // milliseconds, and appending here would put a spinner on screen only to delete it again - + // exactly the flicker the gate exists to remove. The card appears on whichever comes first: + // real content, an error, or LIVE_SUGGESTION_RENDER_DELAY_MS of waiting, which is what keeps + // a genuinely slow answer from looking like a dropped one. + let renderTimer: NodeJS.Timeout | null = setTimeout(() => { + renderTimer = null; + this.appendSuggestion(timestamp, suggestion, epoch); + }, LIVE_SUGGESTION_RENDER_DELAY_MS); + + // Every write after the first goes through here, so the pending card can never be scheduled + // into existence after the sentinel has already decided there is nothing to show. + const publish = (): void => { + if (renderTimer) { + clearTimeout(renderTimer); + renderTimer = null; + } + this.appendSuggestion(timestamp, suggestion, epoch); + }; + + // Refresh a card that is already on screen, without bringing one into existence. Used for + // state changes that carry nothing for the candidate to read. + const refresh = (): void => { + if (this.suggestions.has(timestamp)) { + this.appendSuggestion(timestamp, suggestion, epoch); + } + }; // A plain resettable timer, not a race against reader.read(): a losing read promise stays // pending and has already consumed a read request, so looping would leave two outstanding @@ -93,6 +124,7 @@ class LiveSuggestionService { context: interviewConfig.context, transcripts: transcripts.slice(-TRANSCRIPT_UPLOAD_LIMIT), mode, + turn_verdict: turnVerdict, }; armStallTimer(LIVE_SUGGESTION_TTFB_MS); @@ -109,8 +141,11 @@ class LiveSuggestionService { // check below only fires for Loading, and no timeout rescues it because the stream // ended rather than stalled. An upstream that emits only a block reaches here // with nothing to yield, since _strip_think_stream swallows the whole buffer. + // Refresh, not publish. The response headers land before the backend's gate has decided + // anything - it holds the body back, not the response - so publishing here would render + // the exact card the delay above exists to withhold. suggestion.state = SuggestionState.Loading; - this.appendSuggestion(timestamp, suggestion, epoch); + refresh(); try { while (true) { @@ -122,7 +157,7 @@ class LiveSuggestionService { suggestion.answer += chunk; // Update the suggestion - this.appendSuggestion(timestamp, suggestion, epoch); + publish(); } } @@ -135,7 +170,7 @@ class LiveSuggestionService { } else { suggestion.state = SuggestionState.Success; } - this.appendSuggestion(timestamp, suggestion, epoch); + publish(); } } finally { // releaseLock alone does not cancel the body. Undici documents that an unconsumed, @@ -157,6 +192,9 @@ class LiveSuggestionService { if (aborted && !stalled) { // Superseded by a newer question. Expected, not a failure. suggestion.state = SuggestionState.Stopped; + // Refresh: a card superseded before it was ever shown has no partial answer on it, and + // a Stopped ghost appearing for a question the candidate never saw asked reads as a bug. + refresh(); } else { if (!aborted) { console.error('[LiveSuggestionService] Failed to generate suggestion:', error); @@ -165,15 +203,20 @@ class LiveSuggestionService { suggestion.error = stalled ? 'The response timed out. Please try again.' : getSuggestionErrorMessage(error); + // Publish: a failure is always worth surfacing, even one that failed fast. + publish(); } - this.appendSuggestion(timestamp, suggestion, epoch); } finally { if (stallTimer) clearTimeout(stallTimer); + if (renderTimer) clearTimeout(renderTimer); this.abortMap.delete(taskId); } } - async startGenerateSuggestion(transcripts: Transcript[]): Promise { + async startGenerateSuggestion( + transcripts: Transcript[], + turnVerdict: RequestTurnVerdict = RequestTurnVerdict.Uncertain + ): Promise { // Remove trailing SELF transcripts (same logic as Python) const filteredTranscripts = [...transcripts]; while ( @@ -198,7 +241,7 @@ class LiveSuggestionService { // generateSuggestion owns the abort-map cleanup in its own finally, but it can throw before // reaching the try that guards it - the config and state reads sit above it. Deleting the // entry here on a synchronous rejection keeps a dead controller from being aborted forever. - this.generateSuggestion(taskId, controller, filteredTranscripts).catch((error) => { + this.generateSuggestion(taskId, controller, filteredTranscripts, turnVerdict).catch((error) => { console.error('[LiveSuggestionService] generateSuggestion rejected:', error); this.abortMap.delete(taskId); }); diff --git a/src/main/services/transcript.service.ts b/src/main/services/transcript.service.ts index 24192d7..5de7505 100644 --- a/src/main/services/transcript.service.ts +++ b/src/main/services/transcript.service.ts @@ -1,12 +1,35 @@ import { + INTERVIEWER_TURN_SETTLE_MS, LIVE_SUGGESTION_GAP_MS, SELF_PARTIAL_STALE_MS, TRANSCRIPT_INTER_TRANSCRIPT_GAP_MS, } from '../consts.js'; import { Speaker, Transcript } from '../types/app-state.js'; +import { RequestTurnVerdict } from '../types/llm.js'; +import { classifyInterviewerTurn, TurnVerdict } from '../utils/interviewer-turn.js'; import { appStateService } from './app-state.service.js'; import { liveSuggestionService } from './suggestion-live.service.js'; +/** + * Every trailing `Other` entry in `cleaned` since the candidate last spoke, oldest first. + * + * A pause longer than TRANSCRIPT_INTER_TRANSCRIPT_GAP_MS keeps two interviewer blocks separate in + * `cleaned` even though the candidate never took the floor between them + * ("Tell me about your Kafka work." "Okay?"). Classifying only the last block would read + * that as pure backchannel and silently drop the question in the block before it, so "the turn" is + * every trailing `Other` block, not just the most recent one. + * + * Exported standalone so the concatenation itself is unit-testable against a plain `cleaned` + * array, without driving real wall-clock gaps through `ingest()`. + */ +export function selectTrailingOtherTurn(cleaned: Transcript[]): Transcript[] { + const trailing: Transcript[] = []; + for (let i = cleaned.length - 1; i >= 0 && cleaned[i].speaker === Speaker.Other; i--) { + trailing.unshift(cleaned[i]); + } + return trailing; +} + class TranscriptService { private isActive = false; @@ -15,6 +38,9 @@ class TranscriptService { private otherTranscripts: Transcript[] = []; private otherPartialTranscript: Transcript | null = null; + private turnSettleTimer: NodeJS.Timeout | null = null; + private latestCleaned: Transcript[] = []; + async ingest(channelRaw: string, typeRaw: string, textRaw: string): Promise { if (!this.isActive) return; @@ -80,35 +106,90 @@ class TranscriptService { } } - const lastSelf = cleaned.filter((t) => t.speaker === Speaker.Self).slice(-1)[0]; + // Read by the settle timer, which fires after this call has returned and must see the turn as + // it stands then, not as it stood when the timer was armed. + this.latestCleaned = cleaned; + if (transcript.speaker === Speaker.Other && transcript.isFinal) { - // These two conditions are the only ways a suggestion is silently suppressed, and - // neither surfaces anywhere. Logged so a field or local repro can distinguish - // "the request was never made" from "the request was made and stalled". - - // A partial that has gone quiet is almost certainly orphaned by a dropped ASR socket - // whose final never arrived. Treat it as absent rather than gating indefinitely. - const blockedByPartial = - !!this.selfPartialTranscript && - now - this.selfPartialTranscript.endTimestamp <= SELF_PARTIAL_STALE_MS; - const selfAgeMs = lastSelf ? now - lastSelf.endTimestamp : null; - const skipDueToRecentSelf = - !!lastSelf && lastSelf.isFinal && selfAgeMs !== null && selfAgeMs <= LIVE_SUGGESTION_GAP_MS; - - console.info( - `[TranscriptService] suggestion gate: blockedByPartial=${blockedByPartial}` + - ` skipDueToRecentSelf=${skipDueToRecentSelf}` + - ` lastSelfAgeMs=${selfAgeMs ?? 'none'}` - ); - - if (!blockedByPartial && !skipDueToRecentSelf) { - await liveSuggestionService.startGenerateSuggestion(cleaned); - } + this.scheduleSuggestion(cleaned); } appStateService.updateState({ transcripts: cleaned }); } + /** + * Decide what to do with the interviewer turn that just ended. + * + * Runs on the *merged* turn rather than the single final, so a question the ASR split across two + * finals is classified whole. Re-arming on every final is what makes that work: a fragment parks + * on the settle timer, and its continuation replaces the pending decision with one taken on the + * complete sentence. See `selectTrailingOtherTurn` for why "the turn" can span more than one + * merged block. + */ + private scheduleSuggestion(cleaned: Transcript[]): void { + this.clearTurnSettleTimer(); + + const trailingOther = selectTrailingOtherTurn(cleaned); + const verdict = classifyInterviewerTurn(trailingOther.map((t) => t.text).join(' ')); + + console.info(`[TranscriptService] turn verdict=${verdict}`); + + if (verdict === TurnVerdict.Skip) { + // No request and no card. The NO_SUGGESTION_NEEDED sentinel still backs this up for turns + // the lexicon cannot settle, but a turn caught here never reaches the panel at all, so + // nothing flashes on screen and nothing is billed. + return; + } + + if (verdict === TurnVerdict.Answer) { + void this.fireSuggestion(RequestTurnVerdict.Answer); + return; + } + + this.turnSettleTimer = setTimeout(() => { + this.turnSettleTimer = null; + void this.fireSuggestion(RequestTurnVerdict.Uncertain); + }, INTERVIEWER_TURN_SETTLE_MS); + } + + private async fireSuggestion(verdict: RequestTurnVerdict): Promise { + if (!this.isActive) return; + + const cleaned = this.latestCleaned; + const now = Date.now(); + const lastSelf = cleaned.filter((t) => t.speaker === Speaker.Self).slice(-1)[0]; + + // These two conditions are the only ways a suggestion is silently suppressed, and + // neither surfaces anywhere. Logged so a field or local repro can distinguish + // "the request was never made" from "the request was made and stalled". + + // A partial that has gone quiet is almost certainly orphaned by a dropped ASR socket + // whose final never arrived. Treat it as absent rather than gating indefinitely. + const blockedByPartial = + !!this.selfPartialTranscript && + now - this.selfPartialTranscript.endTimestamp <= SELF_PARTIAL_STALE_MS; + const selfAgeMs = lastSelf ? now - lastSelf.endTimestamp : null; + const skipDueToRecentSelf = + !!lastSelf && lastSelf.isFinal && selfAgeMs !== null && selfAgeMs <= LIVE_SUGGESTION_GAP_MS; + + console.info( + `[TranscriptService] suggestion gate: blockedByPartial=${blockedByPartial}` + + ` skipDueToRecentSelf=${skipDueToRecentSelf}` + + ` lastSelfAgeMs=${selfAgeMs ?? 'none'}` + ); + + if (!blockedByPartial && !skipDueToRecentSelf) { + await liveSuggestionService.startGenerateSuggestion(cleaned, verdict); + } + } + + private clearTurnSettleTimer(): void { + if (this.turnSettleTimer) { + clearTimeout(this.turnSettleTimer); + this.turnSettleTimer = null; + } + } + /** * Promote an in-flight partial to a final when its ASR socket drops. * @@ -146,13 +227,18 @@ class TranscriptService { async stop(): Promise { this.isActive = false; + // A timer left armed here would fire a request against a stopped session. `fireSuggestion` + // re-checks `isActive` as well, so this is belt and braces on the cheaper of the two paths. + this.clearTurnSettleTimer(); } clear(): void { + this.clearTurnSettleTimer(); this.selfTranscripts = []; this.selfPartialTranscript = null; this.otherTranscripts = []; this.otherPartialTranscript = null; + this.latestCleaned = []; appStateService.updateState({ transcripts: [] }); } } diff --git a/src/main/types/llm.ts b/src/main/types/llm.ts index baa741b..0ec83eb 100644 --- a/src/main/types/llm.ts +++ b/src/main/types/llm.ts @@ -61,11 +61,25 @@ export enum SuggestionMode { Professional = 'professional', } +/** + * What the local gate concluded about the interviewer's last turn. + * + * Only two of `TurnVerdict`'s three values travel: a `Skip` never becomes a request at all. The + * backend runs its own classifier for `Uncertain` and trusts `Answer`, which is what keeps a model + * call off the path of every plainly answerable question. Mirrors `TurnVerdict` in the backend's + * `app/schemas/suggestion.py`. + */ +export enum RequestTurnVerdict { + Answer = 'answer', + Uncertain = 'uncertain', +} + export interface GenerateLiveSuggestionRequest extends LLMRequest { profile_data: string; context: string; transcripts: Transcript[]; mode: SuggestionMode; + turn_verdict?: RequestTurnVerdict; } // action request reuses live fields but adds image names diff --git a/src/main/utils/interviewer-turn.ts b/src/main/utils/interviewer-turn.ts new file mode 100644 index 0000000..ef4b1c1 --- /dev/null +++ b/src/main/utils/interviewer-turn.ts @@ -0,0 +1,187 @@ +/** + * Deterministic first stage of the "does this interviewer turn need an answer?" gate. + * + * The `NO_SUGGESTION_NEEDED` sentinel is the last stage, not the first. By the time it fires the + * request has already uploaded the profile, the context and up to 60 transcripts, engaged the + * model, and rendered a pending card that then has to be retracted on screen. A turn that is only + * "mhm" costs exactly what a real question costs. This runs in-process on the merged turn text, in + * microseconds, before any of that. + * + * Precision, not recall, is what this stage optimises for. A missed filler costs one request; a + * skipped question costs the candidate the answer they were waiting for, and fails silently + * mid-interview. So `Skip` is returned only when the backchannel lexicon consumes the turn + * *entirely*; anything it does not fully consume falls through for the model to judge. + */ + +export enum TurnVerdict { + /** Pure backchannel or non-speech. No request, no card. */ + Skip = 'skip', + /** A complete question or directive. Generate now, without waiting for the turn to settle. */ + Answer = 'answer', + /** Could be either, or could be half a sentence. Let the turn settle, then generate. */ + Uncertain = 'uncertain', +} + +/** Transcribed non-speech events: `[laugh]`, `(inaudible)`, ``. */ +const NON_LEXICAL = /[[(<][^\])>]*[\])>]/g; + +const TERMINAL_PUNCTUATION = /[.!?]["')\]]*\s*$/; + +/** + * Question and directive openers. Only consulted together with terminal punctuation, so this does + * not have to distinguish "how" mid-sentence from "how" as an opener. + */ +const CUE = + /\b(what|why|how|when|where|who|which|whose|whom|tell me|walk me|walk us|describe|explain|elaborate|give me|talk about|talk me|share|show me|can you|could you|would you|will you|do you|did you|does|are you|is there|was there|have you|had you|were you|should|suppose|imagine|let's|lets|i'd like|i would like|i want you|go ahead and)\b/; + +/** + * Phrases that carry no question and no content. Matched only from the *front* of the turn and + * repeatedly, so "yeah, okay, got it" is consumed while "okay, so how does that scale?" keeps its + * question. A turn is skipped when this consumes all of it. + */ +const BACKCHANNEL_PHRASES = [ + 'that makes a lot of sense', + 'that makes sense', + 'makes a lot of sense', + 'makes sense', + 'that sounds good', + 'that sounds great', + 'sounds good', + 'sounds great', + 'thank you so much', + 'thank you very much', + 'thanks a lot', + 'thank you', + 'thanks', + 'fair enough', + 'of course', + 'no worries', + 'no problem', + 'very good', + 'very nice', + 'very interesting', + 'really interesting', + 'all right', + 'alright', + 'got it', + 'gotcha', + 'i see', + 'i get it', + 'i understand', + 'understood', + 'noted', + 'uh huh', + 'mm hmm', + 'mhm', + 'mmhmm', + 'hmm', + 'hm', + 'mm', + 'um', + 'uh', + 'er', + 'ah', + 'oh', + 'okay', + 'ok', + 'yeah', + 'yep', + 'yup', + 'yes', + 'right', + 'sure', + 'exactly', + 'true', + 'correct', + 'great', + 'good', + 'nice', + 'cool', + 'perfect', + 'awesome', + 'excellent', + 'wonderful', + 'lovely', + 'interesting', + 'wow', + 'definitely', + 'absolutely', + 'indeed', + 'haha', + 'hehe', + // Connectives an interviewer opens on. Harmless to strip from the front, and stripping them is + // what lets "and yeah, okay" reduce to nothing. + 'so', + 'and', + 'but', + 'well', + 'now', + 'then', + 'also', +] as const; + +// Longest first, so "got it" is never consumed one word at a time by a shorter entry. +const BACKCHANNEL_WORDS: string[][] = BACKCHANNEL_PHRASES.map((phrase) => phrase.split(' ')).sort( + (a, b) => b.length - a.length +); + +/** + * Lowercase, drop non-speech events and every punctuation mark except `?`. + * + * The question mark is kept because it is the single strongest completeness signal available: the + * ASR session runs with `format_turns`, so a finished question reliably arrives punctuated. + * Apostrophes are kept so "let's" and "i'd" still match the cue list. + */ +function normalize(text: string): string { + return text + .toLowerCase() + .replace(NON_LEXICAL, ' ') + .replace(/[‘’]/g, "'") + .replace(/[^a-z0-9'?\s]/g, ' ') + .replace(/\s+/g, ' ') + .trim(); +} + +function stripLeadingBackchannel(words: string[]): string[] { + let index = 0; + + while (index < words.length) { + const phrase = BACKCHANNEL_WORDS.find( + (candidate) => + index + candidate.length <= words.length && + candidate.every((word, offset) => words[index + offset] === word) + ); + if (!phrase) break; + index += phrase.length; + } + + return words.slice(index); +} + +/** + * Classify one merged interviewer turn. + * + * @param rawText The interviewer's turn as it will be shown, punctuation intact. + */ +export function classifyInterviewerTurn(rawText: string): TurnVerdict { + const raw = String(rawText ?? '').trim(); + if (!raw) return TurnVerdict.Skip; + + const normalized = normalize(raw); + // Empty only when the turn was entirely non-speech, e.g. "[laugh]" or "(inaudible)". + if (!normalized) return TurnVerdict.Skip; + + const core = stripLeadingBackchannel(normalized.split(' ')); + if (core.length === 0) return TurnVerdict.Skip; + + const coreText = core.join(' '); + + if (coreText.endsWith('?')) return TurnVerdict.Answer; + + // A directive rather than a question ("Walk me through the migration."). Terminal punctuation is + // required as well: without it the turn is most likely a fragment the speaker is still finishing, + // and answering half a question is worse than waiting out the settle window. + if (TERMINAL_PUNCTUATION.test(raw) && CUE.test(coreText)) return TurnVerdict.Answer; + + return TurnVerdict.Uncertain; +} diff --git a/test/interviewer-turn.test.mjs b/test/interviewer-turn.test.mjs new file mode 100644 index 0000000..a81e31f --- /dev/null +++ b/test/interviewer-turn.test.mjs @@ -0,0 +1,65 @@ +/** + * The deterministic half of the suggestion gate. It decides, before any request is built, whether + * an interviewer turn is pure backchannel ("mhm", "yeah, got it") and can be dropped outright. + * + * Both directions matter, and they fail differently. A filler that slips through costs one wasted + * request and a card that flashes and vanishes - annoying, visible, recoverable. A question + * classified as filler produces *nothing at all*, mid-interview, with no error anywhere. That + * asymmetry is why Skip is only returned when the lexicon consumes the whole turn, and why the + * "never skipped" half of this file is the larger one. + * + * The third verdict is the turn-splitting case: an ASR final is an acoustic endpoint, so a + * half-finished question must land on Uncertain and wait rather than be answered as it stands. + */ +import { createChecker, loadMain } from './helpers.mjs'; + +export async function run() { + const { check, failures } = createChecker('interviewer-turn'); + + const { classifyInterviewerTurn, TurnVerdict } = await loadMain('utils/interviewer-turn.js'); + + const skips = (text) => classifyInterviewerTurn(text) === TurnVerdict.Skip; + const answers = (text) => classifyInterviewerTurn(text) === TurnVerdict.Answer; + const waits = (text) => classifyInterviewerTurn(text) === TurnVerdict.Uncertain; + + // Skipped: nothing here for the candidate to answer. + check('single backchannel token', skips('Okay.')); + check('acknowledgement phrase', skips('Got it.')); + check('stacked backchannels', skips('Yeah, yeah, got it.')); + check('praise plus thanks', skips('Great, thanks.')); + check('hyphenated non-word', skips('Mm-hmm.')); + check('connective plus filler', skips('And yeah, okay.')); + check('agreement', skips('Right, exactly.')); + check('non-speech event only', skips('[laugh]')); + check('non-speech with filler', skips('(inaudible) uh')); + check('empty turn', skips('')); + check('whitespace turn', skips(' ')); + check('closing acknowledgement', skips('Perfect, makes sense.')); + + // Answered immediately: a completed question or directive, so the settle wait is skipped and the + // candidate loses no time. + check('bare question', answers('Why?')); + check('wh question', answers('How did you handle retries?')); + check('question behind a backchannel', answers('Okay, so how does that scale?')); + check('punctuated directive', answers('Tell me about your Kafka work.')); + check('directive behind a connective', answers('So walk me through the migration.')); + check('polite request', answers('Could you describe the architecture?')); + check('backchannel that is really a prompt', answers('Okay?')); + + // The critical half: a real question must never be read as filler, however it opens. + check('question opening on praise is not skipped', !skips('Nice, and how did you test it?')); + check('question opening on thanks is not skipped', !skips('Thanks. What broke first?')); + check('short unpunctuated question is not skipped', !skips('Why Kafka')); + check('directive is not skipped', !skips('Walk me through it.')); + check('statement with content is not skipped', !skips('Your role there.')); + check('closing signal is not skipped', !skips('Thank you for your time today.')); + check('one content word is not skipped', !skips('Kafka.')); + + // Fragments: an ASR final that lands mid-sentence has to wait for its continuation rather than + // be answered as a whole question. Terminal punctuation is what separates the two. + check('unterminated directive waits', waits('So tell me about')); + check('unterminated clause waits', waits('And the part where you')); + check('terminated statement with no cue waits', waits('Your role there.')); + + return failures; +} diff --git a/test/run.mjs b/test/run.mjs index 9ded71d..ecb852b 100644 --- a/test/run.mjs +++ b/test/run.mjs @@ -24,6 +24,8 @@ for (const module of [ // reads the same running state through its own copy of window-control. './running-surface.test.mjs', './tools-export.test.mjs', + './interviewer-turn.test.mjs', + './transcript-turn-selection.test.mjs', './suggestion-sentinel.test.mjs', './mac-update-util.test.mjs', ]) { diff --git a/test/transcript-turn-selection.test.mjs b/test/transcript-turn-selection.test.mjs new file mode 100644 index 0000000..8c8c849 --- /dev/null +++ b/test/transcript-turn-selection.test.mjs @@ -0,0 +1,84 @@ +/** + * `selectTrailingOtherTurn` decides how much of `cleaned` counts as "the interviewer's last turn" + * for classification. It has to span every trailing Other entry since the candidate last spoke, + * not just the most recently merged block. + * + * A pause longer than TRANSCRIPT_INTER_TRANSCRIPT_GAP_MS keeps two interviewer blocks separate in + * `cleaned` even with no candidate turn between them - "Tell me about your Kafka work." + * "Okay?" arrives as two Other entries, not one. Classifying only the last one reads a real + * question as pure backchannel and silently drops it: no request, no card, nothing recoverable. + */ +import { createChecker, loadMain } from './helpers.mjs'; + +export async function run() { + const { check, failures } = createChecker('transcript-turn-selection'); + + const { selectTrailingOtherTurn } = await loadMain('services/transcript.service.js'); + const { classifyInterviewerTurn, TurnVerdict } = await loadMain('utils/interviewer-turn.js'); + const { Speaker } = await loadMain('types/app-state.js'); + + const other = (text, timestamp) => ({ + timestamp, + text, + isFinal: true, + speaker: Speaker.Other, + endTimestamp: timestamp, + }); + const self = (text, timestamp) => ({ ...other(text, timestamp), speaker: Speaker.Self }); + + check('empty transcript selects nothing', selectTrailingOtherTurn([]).length === 0); + + check( + 'a single Other block is selected whole', + selectTrailingOtherTurn([other('Tell me about your Kafka work.', 1000)]).length === 1 + ); + + const twoSeparateBlocks = [ + other('Tell me about your Kafka work.', 1000), + other('Okay?', 10_000), // beyond TRANSCRIPT_INTER_TRANSCRIPT_GAP_MS, so a separate entry + ]; + const trailing = selectTrailingOtherTurn(twoSeparateBlocks); + check('two separated interviewer blocks are both selected', trailing.length === 2); + check( + 'selected blocks stay in speaking order', + trailing[0].text === 'Tell me about your Kafka work.' && trailing[1].text === 'Okay?' + ); + + const stopsAtTheCandidatesTurn = [ + other('How did you handle retries?', 1000), + self('I used exponential backoff.', 2000), + other('Okay.', 3000), + ]; + const afterSelf = selectTrailingOtherTurn(stopsAtTheCandidatesTurn); + check( + 'selection stops at the candidate turn, not the whole transcript', + afterSelf.length === 1 && afterSelf[0].text === 'Okay.' + ); + + check( + 'a transcript ending on the candidate selects nothing', + selectTrailingOtherTurn([other('How did you handle retries?', 1000), self('Sure.', 2000)]) + .length === 0 + ); + + // The regression this file exists to pin: a question followed, after a long pause, by a + // one-word acknowledgement. Classifying the last block alone reads "Right." as pure backchannel + // and drops the question that came before it - the exact silent failure the whole gate exists + // to avoid. + const questionThenAck = [other('Tell me about your Kafka work.', 1000), other('Right.', 10_000)]; + const lastBlockOnly = classifyInterviewerTurn(questionThenAck.at(-1).text); + check( + 'the last block alone misreads this as backchannel (documents the bug)', + lastBlockOnly === TurnVerdict.Skip + ); + + const wholeTurn = selectTrailingOtherTurn(questionThenAck) + .map((t) => t.text) + .join(' '); + check( + 'the concatenated turn keeps the question and is answered', + classifyInterviewerTurn(wholeTurn) === TurnVerdict.Answer + ); + + return failures; +}