Repository navigation
feat(harness): World changes on every trial with records before and after and a start-from-seed check - #329
Closed
KarthikAvinashFI wants to merge 8 commits into
Closed
KarthikAvinashFI wants to merge 8 commits into
KarthikAvinashFI wants to merge 8 commits into
Conversation
…fter, a start-from-seed check and unmeasured checks on an unreadable log
…er, start check and unmeasured checks
…ad of reading empty
…ng it whole in every trial's record
Member
Author
|
Folded into #328 with the related work; nothing dropped. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
TH-8879 (model-tools plan M3 to M6): World changes appeared only on trials with actors, showed actions but not the records they changed, and could not tell a trial that changed nothing from one whose log could not be read. A check reading an unreadable log passed as if nothing happened.
What changed
src/fi/alk/harness/world_snapshot.py(new)_alk_*) are skipped; a table with no primary key or over 5,000 rows is named with why; over 100 changes are counted, not listed; a field's text over 2,000 characters is cut with how much was cut. A start digest says whether a trial began from the seeded recordscall_runner.py,chat_call_runner.pyrecorded: falsewith why when it could not be recorded (voice and chat)world_door.py,world/handle.pyhosted_scheduler.pyevidence_unmeasured, not passed; a check that never reads the log still settlesbundle_author_v2.pyBehaviour callouts
Tests
test_world_snapshot.py: diff, internal tables skipped, no-key table named, row cap, change cap, text cut, start differs from seed, unreadable world, rows changed.test_bundle_author_v2.py::test_a_world_without_actors_gets_the_full_action_log;test_chat_call_runner.py::test_a_trial_without_actors_still_seals_world_changes;test_call_runner.py::test_voice_trial_without_actors_seals_world_changesand the not-recorded case;test_world_door.py::test_events_raise_when_the_log_cannot_be_read;test_hosted_scheduler.pyunreadable log is unmeasured, and a check that never reads the log still settles.Live proof
Local stack with production sandbox specs, run
7447899e(Anthropic, in-process returns agent): every trial has World changes, with reads apart from writes, lost answers and faults.That agent keeps its own records in its process, so "Records before and after" reads "No record changed". Records before and after on a world whose database the agent's tools change is not shown live yet: the returns environment with a tools service could not run on this branch's stack (its worker observation comes with the multi-agent harness line), and a database-backed build of the same agent made today (
2b5c93ea) had every tool call fail inside the agent, cause not found in the time box. The records diff, the row and change caps, the start digest and the unreadable and not-recorded lines are covered by the tests above.Stacked on / merge order
feat/TH-8875-actors. Merge feat(harness): actors, one world door for agent and actors, on top of dev #257 first. feat(harness): model tools additions, World changes on every trial, the world_state artifact and contract arg units (TH-8879, TH-8658, TH-8650) #328 (fix/TH-8658-end-state) stacks on this.Linear: TH-8879