Repository navigation
fix(harness): TH-8748 keep a scenario's key when it is renamed - #203
Conversation
…ed rename pass its own duplicate checks
…nario-key # Conflicts: # src/fi/alk/harness/skills/builder/SKILL.md
|
Live check on a hosted build: latest
|
…into fix/TH-8748-keep-scenario-key
… omits are not changes
|
Verified at 088097a, together with future-agi/future-agi#3606 at fbcdad08, on a local stack driven through the builder chat: all 15 cases in the description pass. That includes an environment built before this change, whose 20 rows kept every number, key, name and id across a rename and a re-save. Full suite with the harness extras: no new failures against |
Review: TH-8748 keep a scenario's key on rename (ALK side)Keeping the key on a rename is the right approach. Tests (run on a fresh VM). Fix before merge
Should fix
Merge order: future-agi/future-agi#3606 first. As the PR body notes, whichever of this PR and #180 ( |
… alone without proving again, clear fields with null, journal renames, repair repeated saved keys, count rename refusals, tidy _changes
… list on both backends, clears go through confirmation, writer edits replace in place, used keys are never reused, repeated keys repaired on open and merge, rename folded into accept_scenario, journal keeps siblings, changelog line
|
Thanks. All six are addressed; the PR is updated and the description reflects the final code.
Also from re-testing: a writer's edit by key now replaces its scenario instead of making a |
# Conflicts: # CHANGELOG.md
1. Why
A scenario keeps one key for the life of its environment: the platform pairs runs and history by it. Today a scenario's key comes from its name whenever a submission does not carry one, so renaming a scenario in the builder chat gives it a new key, and the platform treats it as a new scenario, without its history. A rename also tripped the builder's own duplicate checks, which counted the scenario against itself. And any edit, however small, meant resubmitting every field of the scenario.
2. What changed
src/fi/alk/harness/scenario_tools.pyedit_scenario(scenario_key, <only the fields that change>, clear=[...]): applies the change to the kept scenario withedited(a named field is replaced; a mapping such as the persona changes only the entries named, at any depth; a field named inclear, for examplepersona.accent, goes back to its default) and passes the result through the same path assubmit_scenario, so it is proved like a submission. An edit that leaves the scenario as it is, such as a persona sent back with a field left out, repliesNothing changedand points toclearinstead of reporting a replacement. Its schema is submit's properties with nothing required exceptscenario_key, so a persona-only edit is accepted by both the Agent SDK and the gateway. Inaccept_scenario: a replaced or renamed scenario keeps its key and its place; a rename replies<old> renamed to <new>; a new name alone is applied without the gates or the proof (a name onto a taken one is refused before anything runs, a same-name rename changes nothing); an edit by key from a delegated writer replaces its scenario instead of keeping a-2copy; a new scenario never takes a key the environment has used. A writer's rename journals the dropped entry, and recovery removes only that entry.submit_scenariono longer exposesscenario_keyand points atedit_scenariofor kept scenariossrc/fi/alk/harness/scenario.pyunused_key(a key, or the next numbered one when it is taken) andwith_unique_keys: the first holder keeps a key and a repeat gets the next key no scenario holds or has held, so an existingk-2keeps its keysrc/fi/alk/harness/scenarios.pyused_keys: one set per environment of every key it has used, seeded from the keys the platform sends and from every kept key. Merging sub-agents' scenarios repairs repeated keys withwith_unique_keyssrc/fi/alk/harness/hosted_chat_entrypoint.pyedit_scenariois a writing tool and goes through the chat's confirmation check; a field named inclearcounts as a change, so clearing asks like any other change, and the platform-chat refusals apply to it. The used-key set is seeded from the job'sused_scenario_keys. Repeated saved keys are repaired when the suite is openedsrc/fi/alk/harness/skills/builder/SKILL.mdedit_scenarioand only the changed fields; never drop and resubmit a scenarioCHANGELOG.md3. Behavior callouts
submit_scenario(guard_renames).edit_scenarioand the name-only rename do not go through that wrapper, so whichever of the two merges second extends the guard to them.4. Tests
test_an_edit_sends_only_what_changes_and_keeps_the_resttest_the_edits_edit_scenario_describes_pass_its_schema_on_both_backendsclear) validate against the real schema on the Agent SDK and the gatewaytest_an_edit_clears_what_clear_names_and_merges_nested_mappingsclearresets a field; nested mappings merge; other fields staytest_an_edit_and_a_clear_go_through_the_same_confirmation_as_a_submissiontest_a_renamed_scenario_keeps_its_key_and_replaces_the_old_one,test_an_edit_under_the_same_name_keeps_the_key_it_was_kept_withtest_a_rename_keeps_the_scenario_s_place_and_says_it_was_renamed,test_a_replacement_under_the_same_name_keeps_its_placetest_a_new_name_alone_is_applied_without_proving_the_scenario_again,test_renaming_a_scenario_to_its_own_name_changes_nothing,test_an_edit_onto_a_taken_name_is_refused_before_anything_is_proved,test_a_rename_onto_another_scenario_s_name_is_refusedtest_a_writer_s_edit_by_key_replaces_its_scenario_rather_than_copying_it-2copytest_a_new_scenario_never_takes_a_key_another_one_holds_or_one_it_made_up,test_a_new_scenario_never_takes_a_key_the_environment_has_used,test_a_scenario_dropped_in_the_chat_never_lends_its_key_to_a_new_one,test_the_keys_the_platform_has_used_are_never_given_to_a_new_scenariotest_merging_writers_keeps_the_first_holder_of_a_key_and_gives_repeats_their_own,test_merging_never_moves_a_repeat_onto_a_key_another_scenario_holds,test_repeated_saved_keys_are_repaired_when_the_suite_is_openedk-2keeps its keytest_a_writer_s_rename_never_brings_the_old_name_back_from_the_journal,test_a_writer_s_rename_by_edit_scenario_is_journalled,test_a_writer_s_rename_leaves_a_sibling_s_scenario_of_that_name_in_the_journaltest_a_case_only_rename_keeps_the_scenario_on_disktest_meaning_changes_wait_for_a_yes_to_that_call_but_surface_edits_do_not(extended)Each new test fails with its fix removed (13 of the 15 added this round; the other 2 are guards that also held before) and passes with it. The
Nothing changedreply, added after the live test below found the gap, is locked by an assertion intest_an_edit_sends_only_what_changes_and_keeps_the_restthat fails without it.Harness suite (
tests/test_harness.py tests/harness) with the harness extras: 2421 passed and 8 failed here;devhas 2395 passed and the same 8 failures. No new failures.5. Live test
Re-run on 26 Oct on this branch with #3606 (
783e9650), on a chat environment built before this change (20 scenarios), through the builder chat in plain language. Cases 1 and 2 ran on16a1a759, the rest onf276ec50. For every case: the tool calls from the chat sandbox and the platform rows; every row's number, key and id were compared with a snapshot taken before.edit_scenariowithscenario_keyandnameonly; same key, number and row16a1a759)edit_scenariowith the persona minusoccupationchanged nothing yet replied that the scenario was replaced, and the builder then resubmitted it whole. Fixed inf276ec50: that edit now repliesNothing changedand points toclearf276ec50)submit_scenariounder the same key rather thanedit_scenariowithclear; both keep the keyout-of-scope-pizza-delivery-2), row and number 21Seen once, on
16a1a759: after the person answered No to a name change, the builder proposed the same change again throughsubmit_scenario; the confirmation check asked again, so nothing changed without a yes.Earlier run, before the last two rounds of changes:
edit_scenariowith onlyscenario_keyandname; no confirmation; same key and numberedit_scenariowith only the persona; caller-name check passes; nothing else changestestssent; no confirmationedit_scenariowith both fields<key>-2; all keys distinctedit_scenariowith only the name; same key and numberedit_scenario; same key and number; no other persona field lostdev)6. Commands to run the tests
uv sync --extra livekit --extra harness-stores --extra harness-ui --extra hosted-tracing uv run pytest -q tests/test_harness.py -k "key or rename or merging or edit" tests/harness/test_builder_chat.pyLinear
TH-8748