Skip to content

fix(harness): TH-8765 never send the scenario instruction as the caller's opening - #202

Merged
KarthikAvinashFI merged 11 commits into
devfrom
fix/TH-8765-caller-opening
Oct 27, 2026
Merged

KarthikAvinashFI merged 11 commits into
devfrom
fix/TH-8765-caller-opening

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

1. Why

When the simulated person's first message came back empty, or matched a list of agent-like phrases, converse sent the scenario's instruction instead, word for word. The instruction is stage directions for the simulated person ("You contact … You open by asking … then ask …"), so the agent saw the test script and the run was graded on a conversation that never happened. Every chat run uses this path (hosted chat, Retell chat and local runs all go through converse).

Separately, scenario writers marked free-text tool arguments, such as a search query, as fixed, and a check then compared them exactly, so an agent that searched with different but equally valid wording failed.

2. What changed

File Change
src/fi/alk/harness/run/conversation.py The instruction is never sent as the opening. The phrase list that tried to spot an agent-voiced opening is removed: openings are checked by shape only. As a safeguard, one is not sent when it is empty, only an end marker ([DONE], [STUCK]), or the scenario's instruction handed back on whole words of the line (compared after Unicode NFKC, case folding and whitespace collapsing, so a line that only shares a phrase with a short instruction is still sent). Which part the person plays is set by the opening ask itself, which now says plainly that the model is the person making contact, not the agent, and that these are its first words to the agent, in the language it speaks. When an opening is not sent, the person is told so and asked again for what they came to say; if that is no better, they are told again and asked only for a short greeting in their own language. Only if all three come back unusable does the run open with "Hello.", and the person's next turn then reads You opened with: Hello. followed by The agent replied: …, so the person's session always matches what the agent heard. No call is ended for any of this
src/fi/alk/harness/skills/write-scenarios/SKILL.md Free text the agent composes to find something is not fixed; check what came back or what it acted on, not the words it used
tests/test_harness.py Opening tests below; the phrase-list test (test_an_opening_line_in_the_agents_voice_falls_back_to_the_instruction) is replaced by tests that drive converse itself
CHANGELOG.md One line under Unreleased

3. Behavior callouts

  • The scenario instruction never reaches the agent.
  • A retry is never the same question again: each one tells the person their last line was not sent and why.
  • A non-English person opens in their own language: the greeting is written by the person's own model from its persona, not hard-coded. "Hello." is kept only as the last resort so the call never stops for want of an opening; in 60 real openings measured on this head it was never reached (every first ask was usable).
  • No phrase or pattern is matched against what the person says. An opening that is a real sentence is sent as written, in any language; keeping the person in its part is left to the opening ask. Measured on this head with the real simulator model: 60 first openings across 10 agents and 6 languages (English, Turkish, Hindi, Arabic, Japanese, Spanish), every first opening was used on the first ask (none needed a retry, none fell back to "Hello."), and each was read by hand: none is spoken as the agent (openings in scenarios where the agent places the call read as the person answering it). The openings are generated through converse from scenarios of both chat and voice agents.
  • Voice calls are not affected: they open from the scenario's own initial_message or a generated line, a separate path this PR does not touch.
  • No change when the first opening is usable. Old builds and environments are unaffected: this runs at call time and reads nothing they stored.

4. Tests

Every test drives converse with a stand-in for the person that records each message it is sent, so they check behaviour, not internals.

Test Locks
test_the_opening_ask_says_who_speaks_first_and_in_which_language the opening ask names the person's part (the person making contact, not the agent) and asks for their language
test_an_opening_is_held_back_only_when_empty_a_marker_or_the_instruction (15 cases) held back: empty, whitespace, [DONE], [STUCK], the instruction (also with other spacing and case, decomposed accents, a no-space script with a space added, and inside a longer line); sent as written: an English line starting "Let me", Turkish, Chinese, a line that only shares words with a short instruction ("Cancel my order", "hi"), and any line when there is no instruction
test_an_unusable_opening_is_corrected_then_a_greeting_and_the_instruction_is_never_sent (7 cases) first line used; empty then usable; the instruction echoed then usable; echoed twice then a greeting; empty twice then a Turkish greeting; three unusable then "Hello."; each ask after the first says the line was not sent, no ask repeats, and the last asks for a greeting in the person's language
test_openings_in_any_script_or_shape_reach_the_agent_as_the_person_wrote_them (8 cases) whitespace-only, a bare [DONE], a trailing [DONE], Arabic, Chinese, Hindi after an echo and a [STUCK], a long line, a quoted agent phrase mid-line
test_after_a_corrected_opening_the_next_turn_is_only_the_agents_reply after a successful retry the person gets the agent's reply with no note
test_after_an_opening_the_person_did_not_write_their_next_turn_is_told_what_was_said after the "Hello." fallback the person's next message says what it opened with, then the agent's reply
test_an_opening_the_person_wrote_reaches_them_without_a_note the normal path is unchanged

34 cases. Run against dev's conversation.py, 25 fail on behaviour (no import errors) and 9 pass there and stay as guards (openings that were already sent as written).

Full harness suite (tests/test_harness.py tests/harness): 2,428 passed and 8 failed on this branch; dev has 2,395 passed and the same 8 failures. No new failures.

Live checks (local stack, this branch's code in the sandbox)

Re-run on 26 Oct on this head (8436e845): the forced paths below (the retry, the greeting in English, Turkish and Hindi, and the "Hello." fallback in English and Turkish) give the same results, and one more platform call (scenario #2 of the Page Turner Books chat environment) passed with the person's first message "Hello. Do you have the novel 'Dune' in stock, and how much does it cost?". The three runs started from the UI and the 60-opening sample ran on the commit before (0f2716a8), which changed the opening check and its wording.

  • Real simulator model, each branch reached by forcing only the person's reply empty: the retry, the greeting path in English, Turkish ("Merhaba, iyi günler.") and Hindi ("नमस्ते!"), and the "Hello." fallback, where the person's next turn reads "You opened with: Hello." and the conversation continues in the person's language.
  • Three chat runs started from the UI, 1 scenario each, on three environments of two agents, compared with each environment's last run of the same scenario before this change: the first message is the person's own words in each, never the instruction; message counts are the same; scores are equal or higher. No regression seen.
  • 60 first openings generated through converse with the real model (see section 3). This is a sample, not a guarantee.
  • Not seen live: the real model producing an empty or instruction-echoing opening on its own; that path is covered by the forced runs and the tests.

5. Commands to run the tests

uv sync --extra livekit --extra harness-stores --extra harness-ui --extra hosted-tracing
uv run pytest -q tests/test_harness.py -k "opening or unusable or corrected"

Linear

TH-8765

@KarthikAvinashFI KarthikAvinashFI self-assigned this Oct 23, 2026
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Live check on latest dev with #198 and this branch on top: a 20-scenario build sealed, then a 5-conversation run on the same environment.

Check Result
Build completed, 20 scenarios
Run 5 completed, 0 failed
First caller message natural in all 5, e.g. "hey do you guys have gret gatsby in stock atm?"; none carried the scenario instruction
Grades all checkpoints and judged sub-goals passed and match the transcripts; the search check reads "a query matching the customer request", not an exact string

@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review October 23, 2026 10:13
Base automatically changed from fix/TH-8729-validation-lanes-per-cpu to dev October 23, 2026 10:45
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Final check on c5585786 (this branch with latest dev):

Check Result
Full harness suite with the harness extras 2,400 passed, 8 failed; the same 8 fail on dev at the same commit, so no new failures
Opening test (5 cases) every case fails on dev; the 3 natural-phrase cases also fail on this PR's first version, which aborted them
Live: fresh 6-scenario build + 5-call run 5 of 5 completed; every first message is the caller's own words, none is an instruction; every sub-goal grade matches its transcript

The retry and abort paths did not trigger live; the unit test covers them.

@hadarishav

Copy link
Copy Markdown
Contributor

Review: TH-8765 caller opening

The fix does what the ticket asks: scenario.instruction is never sent as the caller's first turn. I'm asking for a couple of small changes before merge.

Tests (run on a fresh VM). tests/test_harness.py + tests/harness/: 2332 passed on this branch vs 2327 on dev. The 76 failures/errors are the same on both branches; they come from the environment (psycopg not installed, process/dispatch tests). The new opening tests: 8/8 passed. The branch merges cleanly with #203.

Fix before merge

  1. The caller's history doesn't match what the agent heard (run/conversation.py, converse). customer is a Stage, so it keeps session_id and history. When the fallback runs, that history holds the bad opening and the repeated OPENING prompt, but the agent only received "Hello.". Every later customer.say(reply) builds on a conversation the agent never had. Either reset the caller session before falling back, or send a correction turn ("You said: X. That was the agent's voice. Say what you, the caller, want."). That correction turn would also fix item 3.
  2. "Hello." is hard-coded English. Before this change, a non-English scenario fell back to its own instruction; now it opens in English. The agent may then answer in English, which can change how the run is graded. Consider a greeting in the persona's language.

Should fix

  1. The retry sends the identical OPENING text, with the failed answer now in context. It will often fail the same way.
  2. _AS_THE_AGENT matches substrings anywhere in the line, so a real caller saying "Hi, let me explain, my order never came" is treated as agent-voiced. That now costs a retry and possibly the "Hello." fallback. Anchoring "let me " to the start of the line would help.
  3. test_an_opening_line_in_the_agents_voice_falls_back_to_the_instruction: its name now describes the opposite behaviour. Rename it or remove it.
  4. The write-scenarios/SKILL.md free-text search argument change is unrelated to TH-8765 and has no test. Fine to ship together, but name it in the commit message.

Per the PR body, the live 5-call check never reached the retry or greeting paths, and items 1–3 are exactly what a stubbed Stage unit test can't show.

Written by Devin

@hadarishav hadarishav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check comments

…etry differently, greet in the caller's language, anchor the agent-voice check (also keeps the write-scenarios free-text search rule)
…le words after normalising, neutral opening wording, behaviour tests through converse, changelog line
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Thanks Rishav. All six are addressed; the PR is at 8436e84 and the description is updated.

  1. Caller history vs what the agent heard: when an opening is not sent, the person is told it was not sent before being asked again, and after the "Hello." fallback their next turn says what they opened with, then the agent's reply. Tests assert the exact messages the caller model receives on each path.
  2. Hard-coded "Hello.": the opening and retry asks now ask for the person's own language, and the last ask is only a short greeting in that language. "Hello." stays as the last resort when three asks come back unusable, so the call never stops. In 60 real openings it was never reached.
  3. Identical retry: each retry says why the last line was not sent and asks for something different (the request, then only a greeting).
  4. _AS_THE_AGENT: removed rather than anchored. Our ALK standards rule out phrase or regex checks on what the person says, so openings are now checked by shape only: empty, only an end marker, or the scenario's instruction handed back (compared on whole words after normalising). The opening ask now states plainly that these are the person's first words to the agent. In 60 real openings across 6 languages, none needed a retry and none, read by hand, took the agent's part. That is a sample, not a guarantee, and the description says so.
  5. Stale test name: that test went with the phrase check; test_an_opening_is_held_back_only_when_empty_a_marker_or_the_instruction covers the shape checks now.
  6. SKILL.md change: kept in this PR and named in the commit message for 7e4e04b and in the description.

@KarthikAvinashFI
KarthikAvinashFI merged commit 1c34e1f into dev Oct 27, 2026
15 checks passed
@KarthikAvinashFI
KarthikAvinashFI deleted the fix/TH-8765-caller-opening branch October 27, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants