Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
03af3d4
feat(harness): World changes on every trial with records before and a…
KarthikAvinashFI Nov 14, 2026
1872522
feat(harness): keep each trial's end-of-run world state and action lo…
KarthikAvinashFI Nov 14, 2026
01a7115
docs(changelog): World changes on every trial, records before and aft…
KarthikAvinashFI Nov 14, 2026
318b282
docs(skills): the builder's summary reads each field's unit from the …
KarthikAvinashFI Nov 14, 2026
50b6cd6
docs(changelog): each trial's end-of-run world state kept as a world_…
KarthikAvinashFI Nov 14, 2026
38e04bb
Merge branch 'feat/TH-8875-actors' into fix/TH-8879-world-changes
KarthikAvinashFI Nov 15, 2026
951cb40
Merge branch 'feat/TH-8875-actors' into fix/TH-8650-tool-contract-view
KarthikAvinashFI Nov 15, 2026
010c0ba
Merge branch 'fix/TH-8879-world-changes' into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
bd81fdd
test(harness): an unreadable action log raises on the read side inste…
KarthikAvinashFI Nov 15, 2026
4e5eac8
Merge branch 'feat/TH-8875-actors' into fix/TH-8879-world-changes
KarthikAvinashFI Nov 15, 2026
2140338
Merge branch 'fix/TH-8879-world-changes' into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
354751b
Merge branch 'feat/TH-8875-actors' into fix/TH-8650-tool-contract-view
KarthikAvinashFI Nov 15, 2026
096aa7f
fix(harness): World changes cut a huge field's text instead of carryi…
KarthikAvinashFI Nov 15, 2026
99ff804
Merge branch 'fix/TH-8879-world-changes' into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
f65c427
Merge branch 'feat/TH-8875-actors' into fix/TH-8879-world-changes
KarthikAvinashFI Nov 15, 2026
464df6a
Merge branch 'feat/TH-8875-actors' into fix/TH-8650-tool-contract-view
KarthikAvinashFI Nov 15, 2026
4163750
Merge branch 'fix/TH-8879-world-changes' into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
549cdd0
Merge branch 'feat/TH-8875-actors' into fix/TH-8879-world-changes
KarthikAvinashFI Nov 15, 2026
66dc11c
Merge branch 'feat/TH-8875-actors' into fix/TH-8650-tool-contract-view
KarthikAvinashFI Nov 15, 2026
9115142
Merge branch 'fix/TH-8879-world-changes' into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
879abbc
chore: merge fix/TH-8650-tool-contract-view into fix/TH-8658-end-state
KarthikAvinashFI Nov 15, 2026
4556bc9
Merge remote-tracking branch 'origin/feat/TH-8875-actors' into local/…
KarthikAvinashFI Nov 15, 2026
adadb7f
fix(harness): a trial's start is read before its pre-run sessions wri…
KarthikAvinashFI Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,17 @@ Post-rc.1 increments on the release branch. Each one adds gates on top of the
`on_tool` trigger waits for the action in full. An actor waiting for a record only it may write is refused when the
scenario is written. A timeout another actor answers reads `answered_by` only once that actor has acted or spoken,
else `limit_blocked` or `unresolved`.
- Each trial's end-of-run world state is kept: its compared tables and its action log, uploaded as a `world_state`
artifact (`futureagi.world-state.v1`), so a fixed check can grade the trial again without calling the agent.
- World changes on every trial with a door, actors or not: every new world is seeded with the full action log, and
a trial whose tools run where no door stands seals `{"recorded": false, "why": "in_process_tools"}`. The runner
snapshots the world's tables at a trial's start and end (read-only, by primary key, `_alk_*` never compared,
tables over 5,000 rows or without a key named in `skipped`, at most 100 changes listed) and seals `records`
and `start` (`seeded`, `differs` with the tables, or `unknown`). `start` is read after the trial's reset and
before its pre-run sessions write; `records` compares the end against the world after them, so pre-run writes
are neither a start that differs nor the trial's changes. A check that reads an unreadable action log
(`EvidenceUnavailable`) is unmeasured, never settled on an empty log; the scenario errors as
`evidence_unmeasured`.
- Two more fault modes: `unavailable` (the world never sees the call; the agent hears 503 at once) and `error`
(the scenario's `status` 400-599 and `message`). The door answers the agent's keyed retry (`Idempotency-Key`)
from its first result, as it does an actor's; a fault before commit releases the key. A write the world commits
Expand Down
32 changes: 5 additions & 27 deletions src/fi/alk/harness/bundle_author_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,7 @@ def __post_init__(self) -> None:

_IGNORED_ARTIFACT_PARTS = {".git", ".venv", "__pycache__", "node_modules"}
_CERTIFIED_BUNDLE_DIRECTORY = "certified-bundle"
# The agent's tool trace, as every world without actors is seeded.
_AGENT_TRACE_DDL = (
"CREATE TABLE IF NOT EXISTS _alk_tool_trace ("
"id bigserial PRIMARY KEY, name text NOT NULL, arguments jsonb NOT NULL, "
"result jsonb, ok boolean NOT NULL, error text, at double precision NOT NULL);\n"
)
# With actors, the world's action log: every action the tool proxy saw, who took it and what came of it.
# The world's action log: every action the door saw, who took it and what came of it.
_TOOL_TRACE_DDL = (
"CREATE TABLE IF NOT EXISTS _alk_tool_trace ("
"id bigserial PRIMARY KEY, name text NOT NULL, arguments jsonb NOT NULL, "
Expand All @@ -162,27 +156,11 @@ def __post_init__(self) -> None:
)


def _sets_trial_state(authoring: Path) -> bool:
"""Whether any scenario has a pre-run session or sets faults: both need the full action log."""
for document in Path(authoring).glob("scenarios/*/scenario.json"):
try:
body = json.loads(document.read_text(encoding="utf-8"))
except (OSError, ValueError):
continue
if isinstance(body, dict) and (body.get("faults") or body.get("pre_run") or body.get("history")):
return True
return False


def _action_log_seed(authoring: Path) -> str:
"""The action log and who may do what in the world; a world without actors, pre-run sessions or faults is seeded exactly
as before."""
listing = load_actors(authoring)
return (
_TOOL_TRACE_DDL + actors_seed(listing)
if listing.actors or _sets_trial_state(authoring)
else _AGENT_TRACE_DDL
)
"""The world's full action log, and who may do what in it: every world records who did what and what came of it,
so World changes can be shown on every trial, actors or not. A world with no actors seeds no actor rows. A bundle
certified before this keeps the table it was seeded with; the door writes either shape."""
return _TOOL_TRACE_DDL + actors_seed(load_actors(authoring))


def _catalog_image(image: str) -> tuple[str, str]:
Expand Down
159 changes: 139 additions & 20 deletions src/fi/alk/harness/call_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
import os
import stat
import tempfile
import threading
from dataclasses import dataclass, field, replace
from datetime import datetime, timezone
from pathlib import Path
Expand Down Expand Up @@ -1189,15 +1190,17 @@ def approvals(log: list[dict[str, Any]], decisions: Sequence[dict[str, Any]] = (

def _state_changes(
runtime: EnvironmentRuntime, bundle_dir: Path, decisions: Sequence[dict[str, Any]] = ()
) -> list[dict[str, Any]]:
) -> list[dict[str, Any]] | None:
"""What each participant attempted and what came of it; None when the action log could not be read, which the
trial seals as unreadable rather than as no change."""
database = _find_postgres_endpoint(runtime)
if database is None:
return []
return None
try:
log = _read_action_log(database.address)
except Exception as exc: # noqa: BLE001 - no state changes, never a failed call
logger.debug("state changes unreadable: %s", type(exc).__name__)
return []
return None
return trial_state_changes(log, load_actors(bundle_dir).by_id(), decisions)


Expand Down Expand Up @@ -2187,6 +2190,99 @@ def _materialize_vertex_adc(
# --- the runner ----------------------------------------------------------------------------


class TrialRecords:
"""The world's records at each trial's start, and the run's seeded start: what World changes compares a trial's
end against, and whether it started clean. Snapshots are read on a connection that cannot write; one that cannot
be read is sealed as unreadable, never a failed trial."""

def __init__(self) -> None:
self.seeded: dict[str, str] | None = None
self._seeded_taken = False
self._starts: dict[Any, tuple[Any, dict[str, Any]]] = {}
self._lock = threading.Lock()

@staticmethod
def _read(dsn: str | None, moment: str) -> Any:
"""The world's records now, or None when they cannot be read."""
from .world_snapshot import snapshot

try:
return snapshot(dsn) if dsn else None
except Exception as exc: # noqa: BLE001 - only the type: a database error can carry the world's password
logger.debug("trial %s records unreadable: %s", moment, type(exc).__name__)
return None

def start(self, key: Any, dsn: str | None) -> None:
"""Read the world's records at a trial's start, after its reset and before its pre-run sessions write theirs:
whether it started from the seed, and what World changes compares its end against. The first trial's start is
the run's seed."""
from .world_snapshot import digest, start_state

taken = self._read(dsn, "start")
start = digest(taken) if taken is not None else None
with self._lock:
if not self._seeded_taken:
self._seeded_taken = True
self.seeded = start
self._starts[key] = (taken, start_state(self.seeded, start))

def rebase(self, key: Any, dsn: str | None) -> None:
"""Read a trial's records again once its pre-run sessions wrote theirs, so World changes shows only what the
trial itself changed. Whether it started from the seed stays as read before them: pre-run sessions differ by
scenario, and are shown apart, under Before the run."""
taken = self._read(dsn, "start")
with self._lock:
if key in self._starts:
self._starts[key] = (taken, self._starts[key][1])

def end(self, key: Any, dsn: str | None) -> dict[str, Any]:
"""The trial's records before and after (`records`) and whether it started from the seed (`start`)."""
from .world_snapshot import diff

with self._lock:
before, start = self._starts.pop(key, (None, {"state": "unknown", "tables": []}))
after = self._read(dsn, "end") if before is not None else None
if before is None or after is None:
return {"records": {"unreadable": True}, "start": start}
# The end state itself is kept apart (`END_STATE_KEY`): stored as its own artifact, never in the transcript.
return {"records": diff(before, after), "start": start, END_STATE_KEY: after}


# What `TrialRecords.end` carries the trial's end snapshot under, for `upload_end_state`.
END_STATE_KEY = "_end_state"
END_STATE_SCHEMA = "futureagi.world-state.v1"


async def upload_end_state(
adapter: Any, scenario_key: str, sealed: dict[str, Any], log: Sequence[Mapping[str, Any]] | None
) -> str | None:
"""Store the world's records at the trial's end and its action log as the trial's `world_state` artifact, so a
fixed check can grade the trial again without calling the agent; pops the snapshot out of `sealed`. Nothing is
stored when the end could not be read."""
end = sealed.pop(END_STATE_KEY, None)
if end is None:
return None
body = {
"schema": END_STATE_SCHEMA,
"tables": {table: list(rows.values()) for table, rows in end.tables.items()},
"skipped": list(end.skipped),
"action_log": [dict(row) for row in log or []],
}
try:
return await adapter.upload_artifact(
json.dumps(body, sort_keys=True, default=str).encode("utf-8"),
kind=ArtifactKind.WORLD_STATE,
scenario_key=scenario_key,
)
except Exception as exc: # noqa: BLE001 - a lost end state never fails the trial
logger.warning("trial end state not stored: %s", type(exc).__name__)
return None


# Sealed instead of World changes when the agent's tools run where the world's door cannot see them.
NOT_RECORDED = {"recorded": False, "why": "in_process_tools"}


class CallRunnerImpl:
"""Satisfies `hosted_scheduler.CallRunner`. See the module docstring for the three
sub-systems this class implements."""
Expand All @@ -2202,6 +2298,7 @@ def __init__(
self._adapter = adapter
self._context = context
self._place_call = place_call
self._records = TrialRecords()
simulator_secret_values = _canonical_simulator_secrets(
context.simulator_provider_secret_values
)
Expand Down Expand Up @@ -2654,11 +2751,18 @@ async def place_with_actors() -> SimulationReport:
with contextlib.suppress(Exception):
_write_trial_state(runtime, [])

world_database = _find_postgres_endpoint(runtime)
records_dsn = world_database.address if world_database is not None else None

async def place_with_world_actors() -> SimulationReport:
# The trial's starting records, after its reset: whether it started from the seed.
await asyncio.to_thread(self._records.start, id(runtime), records_dsn)
if actors is None:
return await place()
# What the actors wrote before T0 is in the world before the call starts.
await actors.replay()
if actors.pre_run:
await asyncio.to_thread(self._records.rebase, id(runtime), records_dsn)
acting = asyncio.create_task(actors.run())
try:
return await place()
Expand Down Expand Up @@ -2810,22 +2914,32 @@ async def _translate_report(
# from transcript prose.
if not calls and case is not None:
calls = _collect_provider_tool_calls(case)
# The world's door saw the agent's tools: a tool proxy in front of its tools service, or the door in its own
# process. Only then can the trial say what changed in the world.
door_seen = (
self._context.evidence_seam is EvidenceSeam.TOOL_TRACE or runtime.endpoints.get("world_door") is not None
)
acted = (
_actor_rows(runtime, self._context.bundle_dir, actor_decisions)
if case is not None
and self._context.evidence_seam is EvidenceSeam.TOOL_TRACE
if case is not None and door_seen
else []
)
changes = (
_state_changes(runtime, self._context.bundle_dir, actor_decisions)
if acted or actor_decisions
else []
)
duplicates = (
_duplicates(runtime)
if case is not None and self._context.evidence_seam is EvidenceSeam.TOOL_TRACE
else []
changes = _state_changes(runtime, self._context.bundle_dir, actor_decisions) if case is not None and door_seen else []
world_database = _find_postgres_endpoint(runtime)
records = (
await asyncio.to_thread(
self._records.end, id(runtime), world_database.address if world_database is not None else None
)
if case is not None and door_seen
else {}
)
if records:
log_rows: list[dict[str, Any]] | None = None
if world_database is not None:
with contextlib.suppress(Exception):
log_rows = await asyncio.to_thread(_read_action_log, world_database.address)
await upload_end_state(self._adapter, scenario_key, records, log_rows)
duplicates = _duplicates(runtime) if case is not None and door_seen else []
# Hand-off records are not the world's actions: marks pair the graded calls with the action log.
graded = graded_calls(calls)
marks = _agent_marks(runtime, self._context.bundle_dir, graded) if acted else []
Expand All @@ -2846,7 +2960,7 @@ async def _translate_report(
recording = combined if combined and Path(combined).is_file() else None
notices = self._fault_notices(runtime, faults, calls)
# A call that ended with no words still says what did not take effect, and what actors did in the world,
# never only logging it.
# never only logging it; a call with words carries World changes, or why they were not recorded.
if result.transcript or notices or acted or actor_decisions or duplicates or took_part.get("call_actors"):
transcript_document = {
"schema_version": "futureagi.call-transcript.v1",
Expand All @@ -2865,18 +2979,23 @@ async def _translate_report(
"summary": summary,
"events": took_part.get("call_actor_events") or [],
}
if acted or actor_decisions:
# Kept out of the tool trace, whose every line is one of the agent's own calls. `decisions` is
# what each actor chose and why (live) or was scheduled to do (mocked); `actions` what the world
# made of it.
if door_seen:
# Every trial the door saw, actors or not: World changes is shown on each. Kept out of the tool
# trace, whose every line is one of the agent's own calls. `decisions` is what each actor chose
# and why (live) or was scheduled to do (mocked); `actions` what the world made of it.
transcript_document["world_actors"] = {
"actions": acted,
"decisions": list(actor_decisions),
"state_changes": changes,
"state_changes": changes or [],
"approvals": self._approvals(runtime, actor_decisions),
"limits": dict(limits or {}),
"usage": list(actor_usage),
**records,
**({"unreadable": True} if changes is None else {}),
}
else:
# The agent's tools run where the world's door cannot see them: said, never shown as no change.
transcript_document["world_actors"] = dict(NOT_RECORDED)
if duplicates:
# The world applied one of the agent's writes twice after its answer was lost: the scenario fails.
transcript_document.setdefault("world_actors", {})["duplicates"] = duplicates
Expand Down
Loading
Loading