Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7fb8051
feat(harness): refuse browser, computer-use and crawler agents
khushalsonawat Nov 2, 2026
15f74f3
test(harness): stop depending on MCP internals in the write-failure test
khushalsonawat Nov 2, 2026
b874c14
fix(harness): address review of the browser agent refusal
khushalsonawat Nov 2, 2026
7e977c6
fix(harness): keep test folders out of the scan and its reuse honest
khushalsonawat Nov 2, 2026
a89d71c
fix(harness): make the second look hold on both loops, and harden the…
khushalsonawat Nov 4, 2026
880f9a2
fix(harness): end the second look, and read what the scan missed
khushalsonawat Nov 5, 2026
fd72686
fix(harness): close a regex literal's bracket at its line end
khushalsonawat Nov 5, 2026
4435f0e
fix(harness): bound record reads across folders, and keep one look pe…
khushalsonawat Nov 6, 2026
d2c3229
fix(harness): keep record rows relative to their folder, and rescan o…
khushalsonawat Nov 6, 2026
134ff1c
fix(harness): ask only the nearest record folders, and drop a flag th…
khushalsonawat Nov 6, 2026
b9bda58
fix(harness): read each record once, skip slow codecs, and give every…
khushalsonawat Nov 10, 2026
35fd0c9
fix(harness): find the codec name whatever bytes follow it, and count…
khushalsonawat Nov 10, 2026
4060c6e
fix(harness): count browser names in source left unread, and read wha…
khushalsonawat Nov 10, 2026
7726ded
fix(harness): decide a test folder without following links, and read …
khushalsonawat Nov 10, 2026
39e5ae8
fix(harness): decode unicode-escape as Python does, and count only wh…
khushalsonawat Nov 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/sdk-smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ jobs:
tests/harness/test_scenario_source_provision_agent.py
tests/harness/test_hosted_entrypoint_provision_retry.py
tests/harness/test_instrumentation_evidence.py
tests/harness/test_agent_type_refusal.py
tests/harness/test_executor_completed_counts.py
tests/test_bump_deployment_values.py
tests/test_release_workflow.py
tests/test_release_pr_step.py
Expand Down
7 changes: 7 additions & 0 deletions src/fi/alk/harness/HOW-IT-WORKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,13 @@ not exist, or — the one that mattered most in practice — *every* tool having
means the arguments were read and then not recorded. Problems are returned **into the
conversation**, so the model corrects and resubmits rather than a bad contract landing.

A browser reading gets one second look first, naming any browser, crawling or
computer-use tool the code check found in the source. So does a chat or voice reading of source
where the check found one. The next submission stands: a browser answer is refused to
`failure.json` and the build exits 78, and a chat or voice answer is saved. The look gives a value
to send back, which only tells apart calls in one model message: those run before the model reads
any result. The rule and its state are `SecondLook` in `agent_type_check.py`. A browser answer to a chat or voice look is refused at once, since that look said browser stops the build. When calls in one message run one after another, or together once the first scan is done, a later call is taken as the answer before the model has read the look.

**How it is changed later** (`amend.py`). The contract is not frozen, but every change is
recorded with a reason in `amendments[]`, so months later you can still tell what came from the
agent and what came from us:
Expand Down
692 changes: 692 additions & 0 deletions src/fi/alk/harness/agent_type_check.py

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion src/fi/alk/harness/amend.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,8 @@ def set_modality(
False,
f"{named!r} is not a modality. It is one of: {', '.join(known_modalities())}",
)
if named == "browser":
return False, "browser agents aren't supported, so the type stays " + contract.modality + "."
if not why.strip():
return False, "say why: modality decides how every scenario is run"
if named == contract.modality:
Expand All @@ -174,7 +176,6 @@ def set_modality(
"voice": "a call placed to the agent where it is hosted, its own tools answered over a "
"webhook",
"chat": "a typed conversation, the agent reconstructed here from its contract",
"browser": "a browser the agent drives",
}[named]
return True, f"modality is now {named}. Every scenario will be run as {reached}."

Expand Down
6 changes: 6 additions & 0 deletions src/fi/alk/harness/authoring_entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@
import sys
import tempfile

from .agent_type_check import EXIT_REFUSED, failure_message
from .cli import _auto
from .failure_file import FAILURE_FILE
from .job import HarnessJob, ProviderExecutionMode, SourceKind
from .hosted_compose_bootstrap import initialize_hosted_compose
from .provider_import import inspect_provider_target
Expand Down Expand Up @@ -552,6 +554,10 @@ def _await_harness(output: Path) -> int:
finished = output / ".futureagi-build-finished.json"
deadline = time.monotonic() + int(os.environ.get("ALK_HOSTED_AUTHORING_MAX_DURATION_SECONDS") or 3300)
while time.monotonic() < deadline:
# The failure wins: a build that was refused and also finished is still refused.
if (output / FAILURE_FILE).is_file():
print(failure_message(output), file=sys.stderr)
return EXIT_REFUSED
if finished.is_file():
return 0
time.sleep(5)
Expand Down
4 changes: 4 additions & 0 deletions src/fi/alk/harness/chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
from .backends import SessionSpec
from .config import artifact_dir
from .contract import AgentContract
from .failure_file import FAILURE_FILE
from .job import HarnessJob
from .run import stage as run_stage
from .session import Stage
Expand Down Expand Up @@ -152,6 +153,9 @@ async def _open(self, stage_name: str) -> str:
"cannot re-read the agent without knowing where it lives"
)
if stage_name == UNDERSTAND:
# A person's own new reading starts clean.
if self.out:
(self.out / FAILURE_FILE).unlink(missing_ok=True)
self.stage, _ = understand_stage.open_stage(
self.source, out=self.out, ask=self.ask
)
Expand Down
21 changes: 20 additions & 1 deletion src/fi/alk/harness/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@
from .build import open_stage as build_stage
from .build import opening as build_opening
from .build import require_buildable
from .agent_type_check import EXIT_REFUSED, failure_message
from .chat import open_conversation
from .config import (
artifact_dir,
chosen_model,
credentials_hint,
permission_gate,
)
from .failure_file import FAILURE_FILE
from .run.targets import supported as target_kinds
from .scenarios import load as load_written
from .scenarios import open_stage as scenario_stage
Expand Down Expand Up @@ -265,6 +267,10 @@ def after_world(source_root: Any, destination: Path, contract: Any) -> Any:
return export_session_bundle(source_root, destination, name=f"{contract.agent}-environment")


def _report_failure_file(output: Path) -> None:
print(f"\n{failure_message(output)}\n{output / FAILURE_FILE}", file=sys.stderr)


async def _understand(args: argparse.Namespace) -> int:
if args.kind == "provider":
source = resolve(
Expand Down Expand Up @@ -299,17 +305,27 @@ async def _understand(args: argparse.Namespace) -> int:
print(f"model: {chosen_model()}")
print(f"out: {destination}\n")

# A person's own new reading starts clean; a job's failure stands for the platform to read.
if job is None:
(destination / FAILURE_FILE).unlink(missing_ok=True)

await _converse(
stage,
opening(source) + _guidance(args),
interactive=args.interactive,
until=lambda: load(destination) is not None,
until=lambda: load(destination) is not None
or (destination / FAILURE_FILE).is_file(),
nudge=(
"Nothing was saved: you finished without calling submit_contract. Call it now "
"with the contract you worked out."
),
)

if (destination / FAILURE_FILE).is_file():
Comment thread
khushalsonawat marked this conversation as resolved.
Resolved
_report_failure_file(destination)
# A refused reading still cost what its turns did.
print(f"spent: ${stage.spent_usd:.4f}")
return EXIT_REFUSED
Comment thread
khushalsonawat marked this conversation as resolved.
Resolved
contract = load(destination)
if contract is None:
print("\nNo contract was submitted.", file=sys.stderr)
Expand Down Expand Up @@ -1481,6 +1497,9 @@ async def _chat(args: argparse.Namespace) -> int:
await conversation.say(said, on_event=_render_observable)
await conversation.close()
print(f"\nspent: ${conversation.spent_usd:.4f}")
if conversation.out and (conversation.out / FAILURE_FILE).is_file():
_report_failure_file(conversation.out)
return EXIT_REFUSED
return 0


Expand Down
17 changes: 16 additions & 1 deletion src/fi/alk/harness/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import subprocess
from typing import Callable, Protocol

from .failure_file import FAILURE_FILE
from .job import (
FailureDomain,
HarnessFailure,
Expand Down Expand Up @@ -269,7 +270,11 @@ async def run(
retryable=False,
)
status = 1
failure = _failure_from_events(output) if status not in (0, 2) else None
failure = (
_failure_from_file(output) or _failure_from_events(output)
if status not in (0, 2)
else None
)
if generic_failure is not None:
failure = generic_failure
return HarnessJobStatus(
Expand Down Expand Up @@ -337,6 +342,16 @@ def _scenario_count(output: Path) -> int:
return len(value) if isinstance(value, list) else 0


def _failure_from_file(output: Path) -> HarnessFailure | None:
"""The failure a build left for the platform, which says more than its stage event can."""
try:
return HarnessFailure.model_validate_json(
(output / FAILURE_FILE).read_text(encoding="utf-8")
)
except (OSError, ValueError):
return None


def _failure_from_events(output: Path) -> HarnessFailure:
failed: dict = {}
path = output / "harness-events.jsonl"
Expand Down
62 changes: 62 additions & 0 deletions src/fi/alk/harness/failure_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
"""The failure a build leaves for the platform before it exits non-zero."""

from __future__ import annotations

import json
import os
import tempfile
from pathlib import Path

from .job import HarnessFailure

FAILURE_FILE = "failure.json"


def _write_in_place(path: Path, body: str) -> None:
Comment thread
khushalsonawat marked this conversation as resolved.
Resolved
"""For a disk with no hard links: still first-wins, but a reader could catch it part-written.

A disk that refuses ``chmod`` keeps the umask's mode: a whole file beats none.
"""
try:
descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
except FileExistsError:
return
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as file:
file.write(body)
except OSError:
path.unlink(missing_ok=True)
raise
try:
os.chmod(path, 0o644)
except OSError:
pass


def write_failure_file(failure: HarnessFailure, output: Path) -> Path:
"""Write ``failure``. The first failure of a build stands.

Whole, in one step, on a disk with hard links. On one without, it's written in place, where a
reader could catch it part-written.
"""
output = Path(output)
Comment thread
khushalsonawat marked this conversation as resolved.
Resolved
output.mkdir(parents=True, exist_ok=True)
path = output / FAILURE_FILE
body = json.dumps(failure.model_dump(mode="json"), indent=2, sort_keys=True) + "\n"
temporary = tempfile.NamedTemporaryFile(
"w", dir=output, prefix=".failure-", suffix=".tmp", encoding="utf-8", delete=False
)
try:
with temporary:
temporary.write(body)
try:
os.chmod(temporary.name, 0o644)
os.link(temporary.name, path)
except FileExistsError:
pass
except OSError:
# No file reads as a crash, which pays for another full run to reach this same failure.
_write_in_place(path, body)
finally:
os.unlink(temporary.name)
return path
21 changes: 21 additions & 0 deletions src/fi/alk/harness/hosted_chat_entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
tool_server,
)
from .authoring_entrypoint import _load_provider_import_profile, agent_source_kind
from .failure_file import FAILURE_FILE
from .understand import PROVIDER_IMPORT_PROFILE_PATH_ENV
from .chat_policy import EffectiveChatPolicy, apply_chat_policy, load_chat_policy
from .hosted_conversation import (
Expand Down Expand Up @@ -1030,6 +1031,14 @@ def _control_only(self) -> bool:
capabilities = self.client.turn_context.get("capabilities") or {}
return bool(capabilities.get("control_only"))

def _build_is_over(self) -> bool:
"""A control-only build conversation has nothing left to build: it was refused, or its job ended."""
if not self._control_only():
return False
# Not "completed": the post-build check is a platform command on a completed job.
status = self.client.turn_context.get("status") or {}
return status.get("state") in {"failed", "canceled"} or (self.workspace / FAILURE_FILE).is_file()

def _side_chat(self) -> bool:
capabilities = self.client.turn_context.get("capabilities") or {}
return self._control_only() and capabilities.get("builder_session") is False
Expand Down Expand Up @@ -1193,6 +1202,8 @@ async def _carry_on(self) -> bool:
finish_build: when it has saved something that opens more tools, it is reopened with
them and resumes; otherwise it is told the build is not finished. Waits on the person
while a question is open, and gives up after a few turns that change nothing."""
if self._build_is_over():
return False
if not self._control_only() or self._side_chat() or (self.workspace / _BUILD_FINISHED).is_file():
return False
if (self.workspace / _PENDING_QUESTION).is_file():
Expand Down Expand Up @@ -1368,6 +1379,16 @@ async def _handle_command(self, command: dict[str, Any]) -> None:
acknowledge_through=sequence,
)
return
# A restart can redeliver the build command to a fresh sandbox after its job has ended.
if self._from_platform and self._build_is_over():
await self.client.emit(
"turn_completed",
stage=self.conversation.stage_name,
payload={"command_message_id": message_id, "ignored": "the build is over"},
acknowledge_through=sequence,
)
self._journal_unsuccessful(journal, message_id)
return
resume_invocation_id: str | None = None
if state == "started":
resume_invocation_id = (
Expand Down
16 changes: 11 additions & 5 deletions src/fi/alk/harness/skills/understand-agent/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,9 @@ Find, in roughly this order:
`system_prompt_from`, every part it is assembled from: it is copied into the contract as written,
and every later stage decides what the agent should do from it.

7. **The modality.** How a person reaches this agent: a voice session, a text interface, or a
browser it drives. This decides how it is later run, so getting it wrong reroutes every test.
7. **The modality.** How this agent is run: a voice session, a text conversation, or work it
carries out on websites or a computer. This decides how it is later run, so getting it wrong
reroutes every test.
**Decide it from the source and always record it.** A run is unattended, so there is nobody to
ask, and a modality left unsaid is read as chat: a voice agent then never places a call and the
whole run tests nothing. Read it off what the agent depends on:
Expand All @@ -124,7 +125,12 @@ Find, in roughly this order:
name, an STT or TTS provider, an audio session it enters on connect.
- **chat** if a person reaches it as text: an HTTP endpoint taking messages, a completions-shaped
API, a socket carrying turns.
- **browser** if it drives a page rather than being talked to.
- **browser** if a person uses it to carry out work on websites or a computer rather than for
a conversation: driving a browser, operating a computer or crawling sites is its main job.
Ask: does a person come to this agent for a conversation, or to have it carry out that work?
An agent that looks things up on the web to answer a person (research, comparison, support)
is chat, or voice if it takes calls, even if it can't work without that tool. So is one with a
web page in front of it. A coding agent is chat.

When an agent genuinely ships more than one runtime, take the one its entrypoint starts, and say
in `notes` which others exist and that you chose by entrypoint. Never leave the field out.
Expand Down Expand Up @@ -307,8 +313,8 @@ directory. **List what you found and ask which one is being tested.** Do not pic
contract for the wrong one wastes every stage after it, and the person who pointed you here knows
which they meant.

**One agent with several runtimes.** The same tools reachable over voice, over chat, or through a
browser. That is one agent, and what to ask about is the modality, not which agent.
**One agent with several runtimes.** The same tools reachable over voice or over chat. That is one
agent, and what to ask about is the modality, not which agent.

How to tell them apart: look for repeated structure. Several directories that each define their own
set of tools, their own instructions and their own data are several agents. Several entry points
Expand Down
Loading
Loading