Repository navigation
feat(harness): model tools additions, World changes on every trial, the world_state artifact and contract arg units (TH-8879, TH-8658, TH-8650) - #328
Draft
KarthikAvinashFI wants to merge 23 commits into
Conversation
…fter, a start-from-seed check and unmeasured checks on an unreadable log
…g as a world_state artifact
…er, start check and unmeasured checks
…contract's arg_units
…ad of reading empty
…ng it whole in every trial's record
# Conflicts: # tests/harness/test_world_snapshot.py
KarthikAvinashFI
changed the base branch from
fix/TH-8879-world-changes
to
feat/TH-8875-actors
November 15, 2026 19:54
…te, so they never read as a start that differs or as the trial's changes
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR covers
One PR for the model tools additions on top of #257: World changes on every trial with a start-from-seed check, each trial's end-of-run world state kept as a
world_stateartifact, and the builder's summary reading units fromarg_units.Merge order and base
feat/TH-8875-actors(this PR's base). Merge it first.#329: feat(harness): World changes on every trial with records before and after and a start-from-seed check
Why
TH-8879 (model-tools plan M3 to M6): World changes appeared only on trials with actors, showed actions but not the records they changed, and could not tell a trial that changed nothing from one whose log could not be read. A check reading an unreadable log passed as if nothing happened.
What changed
src/fi/alk/harness/world_snapshot.py(new)_alk_*) are skipped; a table with no primary key or over 5,000 rows is named with why; over 100 changes are counted, not listed; a field's text over 2,000 characters is cut with how much was cut. A start digest says whether a trial began from the seeded recordscall_runner.py,chat_call_runner.pyrecorded: falsewith why when it could not be recorded (voice and chat)world_door.py,world/handle.pyhosted_scheduler.pyevidence_unmeasured, not passed; a check that never reads the log still settlesbundle_author_v2.pyBehaviour callouts
Tests
test_world_snapshot.py: diff, internal tables skipped, no-key table named, row cap, change cap, text cut, start differs from seed, unreadable world, rows changed.test_bundle_author_v2.py::test_a_world_without_actors_gets_the_full_action_log;test_chat_call_runner.py::test_a_trial_without_actors_still_seals_world_changes;test_call_runner.py::test_voice_trial_without_actors_seals_world_changesand the not-recorded case;test_world_door.py::test_events_raise_when_the_log_cannot_be_read;test_hosted_scheduler.pyunreadable log is unmeasured, and a check that never reads the log still settles.Live proof
Local stack with production sandbox specs, run
7447899e(Anthropic, in-process returns agent): every trial has World changes, with reads apart from writes, lost answers and faults.That agent keeps its own records in its process, so "Records before and after" reads "No record changed". Records before and after on a world whose database the agent's tools change is not shown live yet: the returns environment with a tools service could not run on this branch's stack (its worker observation comes with the multi-agent harness line), and a database-backed build of the same agent made today (
2b5c93ea) had every tool call fail inside the agent, cause not found in the time box. The records diff, the row and change caps, the start digest and the unreadable and not-recorded lines are covered by the tests above.#328: feat(harness): keep each trial's end-of-run world state and action log as a world_state artifact
Why
TH-8658: after a trial the world's end state was gone with the sandbox. To re-check a trial, or grade it again later, the records it ended with and its action log have to be kept.
What changed
src/fi/alk/harness/call_runner.py,chat_call_runner.pyworld_stateartifact beside the trial's other artifactssrc/fi/alk/harness/outbound.py,hosted_entrypoint.pyCHANGELOG.mdBehaviour callouts
world_state, JSON). A trial whose end could not be read stores none.Tests
tests/harness/test_world_snapshot.py: the trial's end state is stored as its own artifact with the action log; none is stored when the end could not be read.Live proof
Local stack, run
7447899e(6 trials): each trial stored oneworld_stateartifact (schemafutureagi.world-state.v1) holding the world's tables at the end (orders3,refunds1, the seed sentinel) and the trial's action log, for exampleget_orderfaulted then committed,update_order_statusfaulted then committed. The platform keeps the artifact with the trial (future-agi #4128).#327: docs(skills): the builder's summary reads each field's unit from the contract's arg_units
Why
TH-8650: the platform's Contract tab now shows each tool's inputs with their units, read from the contract's
arg_units. The builder's own summary to the person should name the same units, not the person's wording.What changed
src/fi/alk/harness/skills/builder/SKILL.mdarg_units, never from how the person worded itTests
Skill text only; no code path changes.
Live proof
The platform view reading these fields, on a built environment:
User journeys
TrialRecords.startis read after the reset and beforereplay();TrialRecords.rebasere-reads the baseline after the pre-run sessions, only when there are anyTests:
test_world_snapshot.py::test_pre_run_sessions_neither_make_a_trial_read_differs_nor_count_as_its_changesandtest_call_runner.py::test_a_voice_trial_reads_its_start_before_its_pre_run_sessions_write; each fails with the fix reverted.Linear
Linear: TH-8879
Linear: TH-8658
Linear: TH-8650
🤖 Generated with Claude Code
https://claude.ai/code/session_019gWFtEXoqSfwg3LAdyoEUb