Skip to content

feat(harness): model tools additions, World changes on every trial, the world_state artifact and contract arg units (TH-8879, TH-8658, TH-8650) - #328

Draft
KarthikAvinashFI wants to merge 23 commits into
feat/TH-8875-actorsfrom
fix/TH-8658-end-state
Draft

KarthikAvinashFI wants to merge 23 commits into
feat/TH-8875-actorsfrom
fix/TH-8658-end-state

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

What this PR covers

One PR for the model tools additions on top of #257: World changes on every trial with a start-from-seed check, each trial's end-of-run world state kept as a world_state artifact, and the builder's summary reading units from arg_units.

Merge order and base

  1. feat(harness): actors, one world door for agent and actors, on top of dev #257 feat/TH-8875-actors (this PR's base). Merge it first.
  2. This PR, with future-agi #4128.

#329: feat(harness): World changes on every trial with records before and after and a start-from-seed check

Why

TH-8879 (model-tools plan M3 to M6): World changes appeared only on trials with actors, showed actions but not the records they changed, and could not tell a trial that changed nothing from one whose log could not be read. A check reading an unreadable log passed as if nothing happened.

What changed

File Change
src/fi/alk/harness/world_snapshot.py (new) Reads every table of the world's database at a trial's start (after its reset) and end, on a connection that cannot write, and compares them by primary key: records added, changed and removed. Harness tables (_alk_*) are skipped; a table with no primary key or over 5,000 rows is named with why; over 100 changes are counted, not listed; a field's text over 2,000 characters is cut with how much was cut. A start digest says whether a trial began from the seeded records
call_runner.py, chat_call_runner.py Every trial, with or without actors, seals World changes: the full action log, the records before and after, how it started, and recorded: false with why when it could not be recorded (voice and chat)
world_door.py, world/handle.py Reading the action log raises when the log cannot be read, instead of reading empty
hosted_scheduler.py A check that reads an unreadable log is evidence_unmeasured, not passed; a check that never reads the log still settles
bundle_author_v2.py Every world gets the full action log, not only worlds with actors

Behaviour callouts

  • Every trial now carries a World changes record; before, only trials with actors did. Size is bounded by the caps above.
  • A check that read an unreadable log used to pass; it is now unmeasured.
  • A voice call with no words still writes no transcript, so it has no World changes record (unchanged).

Tests

  • test_world_snapshot.py: diff, internal tables skipped, no-key table named, row cap, change cap, text cut, start differs from seed, unreadable world, rows changed.
  • test_bundle_author_v2.py::test_a_world_without_actors_gets_the_full_action_log; test_chat_call_runner.py::test_a_trial_without_actors_still_seals_world_changes; test_call_runner.py::test_voice_trial_without_actors_seals_world_changes and the not-recorded case; test_world_door.py::test_events_raise_when_the_log_cannot_be_read; test_hosted_scheduler.py unreadable log is unmeasured, and a check that never reads the log still settles.
  • Mutation checks: each test fails with its change reverted.

Live proof

Local stack with production sandbox specs, run 7447899e (Anthropic, in-process returns agent): every trial has World changes, with reads apart from writes, lost answers and faults.

Reads, one committed write, its answer lost
A fault, then a committed retry

That agent keeps its own records in its process, so "Records before and after" reads "No record changed". Records before and after on a world whose database the agent's tools change is not shown live yet: the returns environment with a tools service could not run on this branch's stack (its worker observation comes with the multi-agent harness line), and a database-backed build of the same agent made today (2b5c93ea) had every tool call fail inside the agent, cause not found in the time box. The records diff, the row and change caps, the start digest and the unreadable and not-recorded lines are covered by the tests above.

#328: feat(harness): keep each trial's end-of-run world state and action log as a world_state artifact

Why

TH-8658: after a trial the world's end state was gone with the sandbox. To re-check a trial, or grade it again later, the records it ended with and its action log have to be kept.

What changed

File Change
src/fi/alk/harness/call_runner.py, chat_call_runner.py At the end of each trial (chat and voice), the world's records and action log are stored as a world_state artifact beside the trial's other artifacts
src/fi/alk/harness/outbound.py, hosted_entrypoint.py The new artifact kind is uploaded with the trial
CHANGELOG.md Entry

Behaviour callouts

  • One more artifact per trial (world_state, JSON). A trial whose end could not be read stores none.
  • Re-grading a trial from its kept end state is not part of this PR; this only keeps it.

Tests

  • tests/harness/test_world_snapshot.py: the trial's end state is stored as its own artifact with the action log; none is stored when the end could not be read.
  • Mutation check: each test fails with its change reverted.

Live proof

Local stack, run 7447899e (6 trials): each trial stored one world_state artifact (schema futureagi.world-state.v1) holding the world's tables at the end (orders 3, refunds 1, the seed sentinel) and the trial's action log, for example get_order faulted then committed, update_order_status faulted then committed. The platform keeps the artifact with the trial (future-agi #4128).

#327: docs(skills): the builder's summary reads each field's unit from the contract's arg_units

Why

TH-8650: the platform's Contract tab now shows each tool's inputs with their units, read from the contract's arg_units. The builder's own summary to the person should name the same units, not the person's wording.

What changed

File Change
src/fi/alk/harness/skills/builder/SKILL.md When the builder's summary names a field's unit, it reads it from the tool's arg_units, never from how the person worded it

Tests

Skill text only; no code path changes.

Live proof

The platform view reading these fields, on a built environment:

Contract tab, one tool expanded

User journeys

Journey Result Fix
World changes on a scenario whose actors have pre-run sessions The trial's start was read after its pre-run sessions wrote, so it read "Did not start from the seeded records" (pre-run writes differ by scenario) and the pre-run messages counted as the trial's own changes TrialRecords.start is read after the reset and before replay(); TrialRecords.rebase re-reads the baseline after the pre-run sessions, only when there are any
World changes on an actor-free trial Records before and after, and the start check No change needed

Tests: test_world_snapshot.py::test_pre_run_sessions_neither_make_a_trial_read_differs_nor_count_as_its_changes and test_call_runner.py::test_a_voice_trial_reads_its_start_before_its_pre_run_sessions_write; each fails with the fix reverted.

Linear

Linear: TH-8879
Linear: TH-8658
Linear: TH-8650

🤖 Generated with Claude Code

https://claude.ai/code/session_019gWFtEXoqSfwg3LAdyoEUb

…fter, a start-from-seed check and unmeasured checks on an unreadable log
# Conflicts:
#	tests/harness/test_world_snapshot.py
@KarthikAvinashFI KarthikAvinashFI changed the title feat(harness): keep each trial's end-of-run world state and action log as a world_state artifact feat(harness): model tools additions, World changes on every trial, the world_state artifact and contract arg units (TH-8879, TH-8658, TH-8650) Nov 15, 2026
@KarthikAvinashFI
KarthikAvinashFI changed the base branch from fix/TH-8879-world-changes to feat/TH-8875-actors November 15, 2026 19:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant