Repository navigation
fix(harness): TH-8765 never send the scenario instruction as the caller's opening - #202
Conversation
…ning, and keep free-text queries out of fixed values
|
Live check on latest
|
…into fix/TH-8765-caller-opening
|
Final check on
The retry and abort paths did not trigger live; the unit test covers them. |
Review: TH-8765 caller openingThe fix does what the ticket asks: Tests (run on a fresh VM). Fix before merge
Should fix
Per the PR body, the live 5-call check never reached the retry or greeting paths, and items 1–3 are exactly what a stubbed |
…etry differently, greet in the caller's language, anchor the agent-voice check (also keeps the write-scenarios free-text search rule)
…structural checks only
…le words after normalising, neutral opening wording, behaviour tests through converse, changelog line
|
Thanks Rishav. All six are addressed; the PR is at 8436e84 and the description is updated.
|
1. Why
When the simulated person's first message came back empty, or matched a list of agent-like phrases,
conversesent the scenario's instruction instead, word for word. The instruction is stage directions for the simulated person ("You contact … You open by asking … then ask …"), so the agent saw the test script and the run was graded on a conversation that never happened. Every chat run uses this path (hosted chat, Retell chat and local runs all go throughconverse).Separately, scenario writers marked free-text tool arguments, such as a search query, as
fixed, and a check then compared them exactly, so an agent that searched with different but equally valid wording failed.2. What changed
src/fi/alk/harness/run/conversation.py[DONE],[STUCK]), or the scenario's instruction handed back on whole words of the line (compared after Unicode NFKC, case folding and whitespace collapsing, so a line that only shares a phrase with a short instruction is still sent). Which part the person plays is set by the opening ask itself, which now says plainly that the model is the person making contact, not the agent, and that these are its first words to the agent, in the language it speaks. When an opening is not sent, the person is told so and asked again for what they came to say; if that is no better, they are told again and asked only for a short greeting in their own language. Only if all three come back unusable does the run open with "Hello.", and the person's next turn then readsYou opened with: Hello.followed byThe agent replied: …, so the person's session always matches what the agent heard. No call is ended for any of thissrc/fi/alk/harness/skills/write-scenarios/SKILL.mdfixed; check what came back or what it acted on, not the words it usedtests/test_harness.pytest_an_opening_line_in_the_agents_voice_falls_back_to_the_instruction) is replaced by tests that driveconverseitselfCHANGELOG.md3. Behavior callouts
conversefrom scenarios of both chat and voice agents.initial_messageor a generated line, a separate path this PR does not touch.4. Tests
Every test drives
conversewith a stand-in for the person that records each message it is sent, so they check behaviour, not internals.test_the_opening_ask_says_who_speaks_first_and_in_which_languagetest_an_opening_is_held_back_only_when_empty_a_marker_or_the_instruction(15 cases)[DONE],[STUCK], the instruction (also with other spacing and case, decomposed accents, a no-space script with a space added, and inside a longer line); sent as written: an English line starting "Let me", Turkish, Chinese, a line that only shares words with a short instruction ("Cancel my order", "hi"), and any line when there is no instructiontest_an_unusable_opening_is_corrected_then_a_greeting_and_the_instruction_is_never_sent(7 cases)test_openings_in_any_script_or_shape_reach_the_agent_as_the_person_wrote_them(8 cases)[DONE], a trailing[DONE], Arabic, Chinese, Hindi after an echo and a[STUCK], a long line, a quoted agent phrase mid-linetest_after_a_corrected_opening_the_next_turn_is_only_the_agents_replytest_after_an_opening_the_person_did_not_write_their_next_turn_is_told_what_was_saidtest_an_opening_the_person_wrote_reaches_them_without_a_note34 cases. Run against
dev'sconversation.py, 25 fail on behaviour (no import errors) and 9 pass there and stay as guards (openings that were already sent as written).Full harness suite (
tests/test_harness.py tests/harness): 2,428 passed and 8 failed on this branch;devhas 2,395 passed and the same 8 failures. No new failures.Live checks (local stack, this branch's code in the sandbox)
Re-run on 26 Oct on this head (
8436e845): the forced paths below (the retry, the greeting in English, Turkish and Hindi, and the "Hello." fallback in English and Turkish) give the same results, and one more platform call (scenario #2 of the Page Turner Books chat environment) passed with the person's first message "Hello. Do you have the novel 'Dune' in stock, and how much does it cost?". The three runs started from the UI and the 60-opening sample ran on the commit before (0f2716a8), which changed the opening check and its wording.conversewith the real model (see section 3). This is a sample, not a guarantee.5. Commands to run the tests
uv sync --extra livekit --extra harness-stores --extra harness-ui --extra hosted-tracing uv run pytest -q tests/test_harness.py -k "opening or unusable or corrected"Linear
TH-8765