Skip to content

feat(harness): World changes on every trial with records before and after and a start-from-seed check - #329

Closed
KarthikAvinashFI wants to merge 8 commits into
feat/TH-8875-actorsfrom
fix/TH-8879-world-changes
Closed

KarthikAvinashFI wants to merge 8 commits into
feat/TH-8875-actorsfrom
fix/TH-8879-world-changes

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Why

TH-8879 (model-tools plan M3 to M6): World changes appeared only on trials with actors, showed actions but not the records they changed, and could not tell a trial that changed nothing from one whose log could not be read. A check reading an unreadable log passed as if nothing happened.

What changed

File Change
src/fi/alk/harness/world_snapshot.py (new) Reads every table of the world's database at a trial's start (after its reset) and end, on a connection that cannot write, and compares them by primary key: records added, changed and removed. Harness tables (_alk_*) are skipped; a table with no primary key or over 5,000 rows is named with why; over 100 changes are counted, not listed; a field's text over 2,000 characters is cut with how much was cut. A start digest says whether a trial began from the seeded records
call_runner.py, chat_call_runner.py Every trial, with or without actors, seals World changes: the full action log, the records before and after, how it started, and recorded: false with why when it could not be recorded (voice and chat)
world_door.py, world/handle.py Reading the action log raises when the log cannot be read, instead of reading empty
hosted_scheduler.py A check that reads an unreadable log is evidence_unmeasured, not passed; a check that never reads the log still settles
bundle_author_v2.py Every world gets the full action log, not only worlds with actors

Behaviour callouts

  • Every trial now carries a World changes record; before, only trials with actors did. Size is bounded by the caps above.
  • A check that read an unreadable log used to pass; it is now unmeasured.
  • A voice call with no words still writes no transcript, so it has no World changes record (unchanged).

Tests

  • test_world_snapshot.py: diff, internal tables skipped, no-key table named, row cap, change cap, text cut, start differs from seed, unreadable world, rows changed.
  • test_bundle_author_v2.py::test_a_world_without_actors_gets_the_full_action_log; test_chat_call_runner.py::test_a_trial_without_actors_still_seals_world_changes; test_call_runner.py::test_voice_trial_without_actors_seals_world_changes and the not-recorded case; test_world_door.py::test_events_raise_when_the_log_cannot_be_read; test_hosted_scheduler.py unreadable log is unmeasured, and a check that never reads the log still settles.
  • Mutation checks: each test fails with its change reverted.

Live proof

Local stack with production sandbox specs, run 7447899e (Anthropic, in-process returns agent): every trial has World changes, with reads apart from writes, lost answers and faults.

Reads, one committed write, its answer lost
A fault, then a committed retry

That agent keeps its own records in its process, so "Records before and after" reads "No record changed". Records before and after on a world whose database the agent's tools change is not shown live yet: the returns environment with a tools service could not run on this branch's stack (its worker observation comes with the multi-agent harness line), and a database-backed build of the same agent made today (2b5c93ea) had every tool call fail inside the agent, cause not found in the time box. The records diff, the row and change caps, the start digest and the unreadable and not-recorded lines are covered by the tests above.

Stacked on / merge order

Linear: TH-8879

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Folded into #328 with the related work; nothing dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant