Skip to content

fix(harness): TH-8748 keep a scenario's key when it is renamed - #203

Merged
KarthikAvinashFI merged 18 commits into
devfrom
fix/TH-8748-keep-scenario-key
Oct 27, 2026
Merged

KarthikAvinashFI merged 18 commits into
devfrom
fix/TH-8748-keep-scenario-key

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

1. Why

A scenario keeps one key for the life of its environment: the platform pairs runs and history by it. Today a scenario's key comes from its name whenever a submission does not carry one, so renaming a scenario in the builder chat gives it a new key, and the platform treats it as a new scenario, without its history. A rename also tripped the builder's own duplicate checks, which counted the scenario against itself. And any edit, however small, meant resubmitting every field of the scenario.

2. What changed

File Change
src/fi/alk/harness/scenario_tools.py New edit_scenario(scenario_key, <only the fields that change>, clear=[...]): applies the change to the kept scenario with edited (a named field is replaced; a mapping such as the persona changes only the entries named, at any depth; a field named in clear, for example persona.accent, goes back to its default) and passes the result through the same path as submit_scenario, so it is proved like a submission. An edit that leaves the scenario as it is, such as a persona sent back with a field left out, replies Nothing changed and points to clear instead of reporting a replacement. Its schema is submit's properties with nothing required except scenario_key, so a persona-only edit is accepted by both the Agent SDK and the gateway. In accept_scenario: a replaced or renamed scenario keeps its key and its place; a rename replies <old> renamed to <new>; a new name alone is applied without the gates or the proof (a name onto a taken one is refused before anything runs, a same-name rename changes nothing); an edit by key from a delegated writer replaces its scenario instead of keeping a -2 copy; a new scenario never takes a key the environment has used. A writer's rename journals the dropped entry, and recovery removes only that entry. submit_scenario no longer exposes scenario_key and points at edit_scenario for kept scenarios
src/fi/alk/harness/scenario.py unused_key (a key, or the next numbered one when it is taken) and with_unique_keys: the first holder keeps a key and a repeat gets the next key no scenario holds or has held, so an existing k-2 keeps its key
src/fi/alk/harness/scenarios.py used_keys: one set per environment of every key it has used, seeded from the keys the platform sends and from every kept key. Merging sub-agents' scenarios repairs repeated keys with with_unique_keys
src/fi/alk/harness/hosted_chat_entrypoint.py edit_scenario is a writing tool and goes through the chat's confirmation check; a field named in clear counts as a change, so clearing asks like any other change, and the platform-chat refusals apply to it. The used-key set is seeded from the job's used_scenario_keys. Repeated saved keys are repaired when the suite is opened
src/fi/alk/harness/skills/builder/SKILL.md Edit or rename with edit_scenario and only the changed fields; never drop and resubmit a scenario
CHANGELOG.md One line under Unreleased

3. Behavior callouts

4. Tests

Test Locks
test_an_edit_sends_only_what_changes_and_keeps_the_rest an edit with only a new name renames in place, same key, every other field unchanged; an edit naming one persona field keeps the others; an edit that sends back the saved values with a field left out changes nothing and says so; an unknown key is an error
test_the_edits_edit_scenario_describes_pass_its_schema_on_both_backends the documented inputs (persona-only, clear) validate against the real schema on the Agent SDK and the gateway
test_an_edit_clears_what_clear_names_and_merges_nested_mappings clear resets a field; nested mappings merge; other fields stay
test_an_edit_and_a_clear_go_through_the_same_confirmation_as_a_submission through the chat guard: an edit and a clear ask; the platform chat refuses a fixture clear
test_a_renamed_scenario_keeps_its_key_and_replaces_the_old_one, test_an_edit_under_the_same_name_keeps_the_key_it_was_kept_with renames and edits keep the key
test_a_rename_keeps_the_scenario_s_place_and_says_it_was_renamed, test_a_replacement_under_the_same_name_keeps_its_place place kept; rename reply
test_a_new_name_alone_is_applied_without_proving_the_scenario_again, test_renaming_a_scenario_to_its_own_name_changes_nothing, test_an_edit_onto_a_taken_name_is_refused_before_anything_is_proved, test_a_rename_onto_another_scenario_s_name_is_refused name-only renames skip gates and proof; no-op and taken-name cases
test_a_writer_s_edit_by_key_replaces_its_scenario_rather_than_copying_it a writer's edit by key leaves no -2 copy
test_a_new_scenario_never_takes_a_key_another_one_holds_or_one_it_made_up, test_a_new_scenario_never_takes_a_key_the_environment_has_used, test_a_scenario_dropped_in_the_chat_never_lends_its_key_to_a_new_one, test_the_keys_the_platform_has_used_are_never_given_to_a_new_scenario used keys are never reused
test_merging_writers_keeps_the_first_holder_of_a_key_and_gives_repeats_their_own, test_merging_never_moves_a_repeat_onto_a_key_another_scenario_holds, test_repeated_saved_keys_are_repaired_when_the_suite_is_opened duplicate-key repair; an existing k-2 keeps its key
test_a_writer_s_rename_never_brings_the_old_name_back_from_the_journal, test_a_writer_s_rename_by_edit_scenario_is_journalled, test_a_writer_s_rename_leaves_a_sibling_s_scenario_of_that_name_in_the_journal journal after renames
test_a_case_only_rename_keeps_the_scenario_on_disk a case-only rename on a case-insensitive filesystem keeps the folder
test_meaning_changes_wait_for_a_yes_to_that_call_but_surface_edits_do_not (extended) a rename alone does not ask; a rename that changes the instruction asks

Each new test fails with its fix removed (13 of the 15 added this round; the other 2 are guards that also held before) and passes with it. The Nothing changed reply, added after the live test below found the gap, is locked by an assertion in test_an_edit_sends_only_what_changes_and_keeps_the_rest that fails without it.

Harness suite (tests/test_harness.py tests/harness) with the harness extras: 2421 passed and 8 failed here; dev has 2395 passed and the same 8 failures. No new failures.

5. Live test

Re-run on 26 Oct on this branch with #3606 (783e9650), on a chat environment built before this change (20 scenarios), through the builder chat in plain language. Cases 1 and 2 ran on 16a1a759, the rest on f276ec50. For every case: the tool calls from the chat sandbox and the platform rows; every row's number, key and id were compared with a snapshot taken before.

# Typed in the chat Result
1 "rename scenario 6 to search_books_changes_mind" Pass: edit_scenario with scenario_key and name only; same key, number and row
2 "in scenario 4, the person should be called Meera Iyer instead. nothing else changes", answered yes Pass: confirmation asked (the person's name changes who the caller is); only the name changed; same row
3 "in scenario 9, the person shouldn't have an occupation at all" (on 16a1a759) Occupation cleared on the same row, but the path showed a gap: edit_scenario with the persona minus occupation changed nothing yet replied that the scenario was replaced, and the builder then resubmitted it whole. Fixed in f276ec50: that edit now replies Nothing changed and points to clear
4 the same request on scenarios 14 and 16 (on f276ec50) Pass: occupation cleared, every other field and the row kept. The builder chose submit_scenario under the same key rather than edit_scenario with clear; both keep the key
5 "remove scenario 15, the pizza delivery one", answered yes; then "add a new scenario named out_of_scope_pizza_delivery ..." Pass: the dropped row stays deleted with its key; the new scenario gets its own key (out-of-scope-pizza-delivery-2), row and number 21
6 After all of the above Pass: every original row except the dropped one keeps its number, key and id
7 One call on renamed scenario 6 Pass: graded on row 6 under its original key

Seen once, on 16a1a759: after the person answered No to a name change, the builder proposed the same change again through submit_scenario; the confirmation check asked again, so nothing changed without a yes.

Earlier run, before the last two rounds of changes:

# Typed in the chat Expected Result
1 "rename scenario 2 to something shorter, maybe second_scenario_renamed" edit_scenario with only scenario_key and name; no confirmation; same key and number Pass
2 "the person in scenario 1 should be called Meera Iyer instead, nothing else changes" edit_scenario with only the persona; caller-name check passes; nothing else changes Pass
3 "for scenario 4, change the passes-when so it says the agent repeats the order number back" only tests sent; no confirmation Pass
4 "on scenario 5, set max turns to 12 and add the keyword security" one edit_scenario with both fields Pass
5 add a new scenario under scenario 2's old name a new row with its own key <key>-2; all keys distinct Pass (8 rows, 8 keys, 8 numbers)
6 "rename scenario 3 to two_orders_in_sequence", then add a new one under its old name, then save twice renamed keeps key and number; the new one gets its own row; saves succeed Pass
7 "rename scenario 6 to pizza_redirect and make it about a refund instead", answered no confirmation asked; declined; nothing changes Pass
8 "rename scenario 42 to something_else" says it does not exist; nothing changes Pass
9 "rename scenario 4 to search_catalog_and_track_order" (scenario 1's name) refused or asked; nothing changes Pass (asked which to rename; nothing changed)
10 "the sub-goal that checks order status should also require the agent to tell the customer the carrier", answered yes confirmation; sub-goal updated; dependent scenarios proved again; shown by the environment detail Pass (7 scenarios proved again; new criterion shown)
11 "rename scenario 3 back to track_multiple_orders_sequence_v1" edit_scenario with only the name; same key and number Pass
12 "please save the environment again" after the edits rows, Scenarios tab and saved scenarios show the edited values; keys and numbers unchanged Pass
13 On an environment built before this change: "please rename scenario 5 to a shorter name: cancel_shipped_refusal" same behaviour; its other 19 rows untouched Pass (key and number kept; no row recreated)
14 One call on renamed scenario 3 the call runs under the scenario's original key Pass (graded, key unchanged)
15 Re-save of the environment built before this change no row's number, key, name or id changes Pass (20 rows identical before and after)
16 "in scenario 6, make the caller's accent Australian. nothing else changes" edit_scenario; same key and number; no other persona field lost Pass (key and number kept; no persona field lost. The builder also moved the location to match and said so; a chat environment clears the accent, as on dev)

6. Commands to run the tests

uv sync --extra livekit --extra harness-stores --extra harness-ui --extra hosted-tracing
uv run pytest -q tests/test_harness.py -k "key or rename or merging or edit" tests/harness/test_builder_chat.py

Linear

TH-8748

@KarthikAvinashFI KarthikAvinashFI self-assigned this Oct 23, 2026
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Live check on a hosted build: latest dev with this branch on top, an existing 20-scenario environment, and one builder chat message asking to rename scenario #4 and change nothing else.

Check Result
Builder used rename_scenario, then save_changes with nothing added, changed or removed
Scenario #4 same key and number, new name
Suite 20 rows, 20 distinct keys, 20 distinct numbers; no second row

@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review October 23, 2026 10:33
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Verified at 088097a, together with future-agi/future-agi#3606 at fbcdad08, on a local stack driven through the builder chat: all 15 cases in the description pass. That includes an environment built before this change, whose 20 rows kept every number, key, name and id across a rename and a re-save. Full suite with the harness extras: no new failures against dev.

@hadarishav

Copy link
Copy Markdown
Contributor

Review: TH-8748 keep a scenario's key on rename (ALK side)

Keeping the key on a rename is the right approach. edit_scenario is correctly in _NEEDS_CONFIRMATION and _WRITING_TOOLS, and _changes comparing through edited() fixes the false "changed" signal for omitted defaults. I'm asking for changes before merge.

Tests (run on a fresh VM). tests/test_harness.py + tests/harness/: 2333 passed on this branch. The 76 failures/errors are the same set as dev (environmental). The key, edit and rename tests: 14/14 passed. The branch merges cleanly with #202.

Fix before merge

  1. A rename moves the scenario to the end of the suite, and the tool reply calls it "kept". I confirmed this with a throwaway test: [adds-a-big-mac, orders-again] became [orders-again, renamed-first], and the reply began renamed-first kept. …. accept_scenario drops previous and then calls kept.append(scenario). replaced is computed after previous has been removed, so a rename is reported as a new scenario. The platform keeps the row number (fix(simulate): TH-8748 match published scenarios to rows by key first future-agi#3606), but anything in ALK that uses suite position sees the order change, and the builder model is told a scenario was added. Suggest: insert at previous's index, and report "<old> renamed to <new>".

Should fix

  1. A name-only rename runs every gate and the full proof again. edit_scenario → submit_scenario.handler(edited(held, args)). An older scenario that fails a gate added since it was written (for example a grid declared later) can't be renamed, and the refusal names an unrelated rule. Consider skipping the gates when only name changes, or add a test for that case.
  2. Duplicate saved keys survive merged() (test_merging_writers_never_changes_a_saved_key_or_repeats_one locks this in). On the platform, two scenarios with the same key in one suite make index_scenarios raise IntegrityError on uniq_harness_scenario_key; that is already the case on dev (see #3606). Either repair saved duplicates once here, or make the platform tolerate them.
  3. edited() drops None (exclude_none=True) and merges dictionaries only one level deep, so an edit can't clear a persona field such as an accent. Say so in the tool description.
  4. Possibly unreachable: with persist=False, only the new scenario is journalled. Removing previous isn't recorded, so recovering from the journal after a writer rename could bring back both names.
  5. Nit: in _changes, the final generator reuses key, and args = edited(target, args) rebinds the parameter. Both are harmless but make it harder to read.

Merge order: future-agi/future-agi#3606 first. As the PR body notes, whichever of this PR and #180 (guard_renames) or platform #3533 (plan_pairs) lands second has to carry the key rule into the rewritten code.

Written by Devin

@hadarishav hadarishav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check comments

… alone without proving again, clear fields with null, journal renames, repair repeated saved keys, count rename refusals, tidy _changes
… list on both backends, clears go through confirmation, writer edits replace in place, used keys are never reused, repeated keys repaired on open and merge, rename folded into accept_scenario, journal keeps siblings, changelog line
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Thanks. All six are addressed; the PR is updated and the description reflects the final code.

  1. Rename moves to the end and reads "kept": a replaced or renamed scenario goes back into its own place, and a rename replies " renamed to " (test_a_rename_keeps_the_scenario_s_place_and_says_it_was_renamed, test_a_replacement_under_the_same_name_keeps_its_place).
  2. Name-only rename reruns every gate: a new name alone is applied in accept_scenario without the gates or the proof (the name is still validated and a taken name is refused before anything runs); there is no separate write path. test_a_new_name_alone_is_applied_without_proving_the_scenario_again fails if any gate or the proof runs.
  3. Duplicate saved keys: repaired when the suite is opened and when writers are merged. The first holder keeps the key; a repeat gets the next key that no scenario holds or has held, so an existing k-2 keeps its key. #3606 gives the same key.
  4. edited() and None: an edit can clear a field by naming it in clear (for example persona.accent), and mappings merge at any depth. I used a clear list rather than null because the gateway path drops nullable types from the schema, so null could never validate there; a test checks the documented inputs against the real schema on both paths. Clearing goes through the same confirmation as any other change.
  5. Journal after a writer rename: it was reachable. A rename now journals the dropped entry, and recovery removes only that entry, so a sibling writer's scenario of the same name stays.
  6. Nits in _changes: done.

Also from re-testing: a writer's edit by key now replaces its scenario instead of making a -2 copy, and the keys the platform has already used (including dropped scenarios') are never given to a new scenario.

hadarishav
hadarishav previously approved these changes Oct 27, 2026
@KarthikAvinashFI
KarthikAvinashFI merged commit a175d79 into dev Oct 27, 2026
15 checks passed
@KarthikAvinashFI
KarthikAvinashFI deleted the fix/TH-8748-keep-scenario-key branch October 27, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants