Skip to content

feat(harness): scenario proof honesty and lifecycle (TH-8752, TH-8918, TH-8929, TH-8754) - #321

Draft
KarthikAvinashFI wants to merge 42 commits into
devfrom
feat/TH-8918-scenario-lifecycle
Draft

KarthikAvinashFI wants to merge 42 commits into
devfrom
feat/TH-8918-scenario-lifecycle

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

What this PR covers

One PR for the Create scenarios work on dev: proofs save whether each solution ran, negative controls, the leaked-answer check, grounding and suite proving, partial apply, and restoring scenarios a rebuild dropped.

Merge order and base

  1. dev (this PR's base).
  2. This PR, before or with future-agi #4107, which passes --prove-scenarios and reads scenario-proofs.json.

#320: fix(harness): save whether each proof's solution ran and report proved counts

Why

TH-8752: a scenario whose reference solution never ran (its tools have no endpoint in the prove lane, so the steps were assumed) was saved as holds: true and reported as proved. Plan task T05 ("Make scenarios valid and meaningfully graded") needs an unproved case to read as unproved everywhere it is counted.

This PR makes the saved proof say what really ran, and makes the build's Finished message report proved, kept as written and not proved apart.

What changed

File Change
src/fi/alk/harness/prove.py Proof.proved (holds, nothing assumed, nothing deferred), Proof.kept_as_written (not proved only because the lane could not execute something) and Proof.blocked_reason() (first reason, faults before lane limits). LANE_CODES = proof_deferred, solution_assumed, control_not_run.
src/fi/alk/harness/proofs.py Proof entries are format 2: proved, kept_as_written, assumed, solution_steps, deferred, judged_only, blocked_reason, and a why that names the assumed tools. New proof_key(), unprovable_entry(), proof_counts().
src/fi/alk/harness/hosted_chat_entrypoint.py finish_build writes the counts into the build-finished file and its reply reads "N proved, N kept as written (tools), N not proved (reasons). Report exactly these counts".
src/fi/alk/harness/builder.py, nudge.py, scenario_tools.py Every record_proof call passes solution_steps and saves under proof_key().
src/fi/alk/harness/skills/builder/SKILL.md "Finishing a build": say proved only for the count finish_build reports as proved.
tests/harness/proof_world.py Test world helper for the proof tests; its stub returns the scenario-code outcome the prove step reads.
tests/harness/test_proofs.py, tests/harness/test_builder_chat.py New tests (below).

Behavior callouts

  • A proof that holds but assumed a step is saved holds: true, proved: false, kept_as_written: true, with why naming the tools whose steps never ran. Older readers that only look at holds still see true; the platform side reads proved for format 2 (companion PR below).
  • A deferred proof reads proof_deferred, not not_ready: its gates never ran, so they are unset rather than failed.
  • The closing counts are written by code: when the build finished during the turn, the reply gets "Proof counts: N proved, N kept as written, ..." appended, and a reply that claims more proved than the count gets a correction line.
  • No change to which scenarios are kept or refused.

Tests

  • 6 new tests: test_assumed_steps_are_saved_and_not_proved, test_proof_counts_split_kept_as_written, test_unprovable_is_format_two, test_deferred_reads_proof_deferred_not_not_ready, test_holds_failures_map_to_codes, test_finished_message_reports_counts.
  • Run module by module on the stacked lifecycle head (which contains this branch): the touched modules plus every tests/harness module importing changed code (35 modules) and tests/test_harness.py. All new and named tests pass. Failures that exist identically on dev (0203bdf5): test_harness.py 3, test_background_noise.py 2, test_grade.py 1, test_harness_architecture.py 11.
  • Mutation checks across this PR and the stacked lifecycle PR: 32 of 32 mutations killed.

Added on this head:

  • b9b21ee6: the finishing reply carries the code-written counts and corrects an overclaim. Tests test_finished_turn_appends_code_counts, test_overclaim_gets_a_correction, test_exact_claim_gets_no_correction.
  • 99122c5d: test_assumed_steps_are_saved_and_not_proved failed on this branch alone (the stub returned the check outcome type, and the check credited a failed call); the stub now returns the scenario-code outcome and the check reads only calls that succeeded.
  • This branch alone, the targeted harness modules in one process: 2592 pass; three failures, the same on dev (0203bdf5): test_hosted_claude_session_uses_durable_resume_and_restricted_tools, test_a_skill_only_names_tools_its_stage_actually_has, test_the_system_prompt_stays_within_budget.

Live proof

Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress). Validations re-proved seven existing environments with this code.

Case Result Proof
Cases whose tools run only inside the agent Saved proved: false, assumed tools listed, counted "kept as written", never proved Toolbar line below
Judged-only cases on a voice agent Proved, counted proved Second image
Finished message counts on a fresh build Unit only (test_finished_message_reports_counts); no fresh full build was run on this head Not shown

Counts line: 0 proved, 9 kept as written, 0 not proved on an environment whose cases cannot run their solutions

After validation: 5 proved, 0 kept as written, 1 not proved

Playwright e2e

None in this repo. The platform flows that read these proofs (SIM-E2E-001 to 006 and 008) passed 7 of 7 on the stacked platform lifecycle branch, before its last base merge.

Scope out

  • Proving the reference solution on the real stack, so that agent-bound tools actually run (they stay "kept as written").
  • CLI and SDK display of the counts: later, after the platform contract is released.

#321: feat(harness): negative controls, leaked-answer check, grounding and suite proving

Why

TH-8918 (scenario lifecycle), plan task T05 ("Make scenarios valid and meaningfully graded"): a scenario is only worth running when its checks can fail a wrong behaviour, when nothing the caller or the agent sees gives the answer away, and when what it cites really says what it claims. Before this PR none of the three was checked: a case passed with checks that also passed a wrong refund, a case could quote its own grader text in the caller's instruction, and sources were not recorded at all.

This PR adds the three checks to the prove gate and a suite-wide re-prove that the platform's Validate runs at certify. The platform side (lifecycle states, Validate, archive, restore, source changes) is future-agi feat/TH-8918-scenario-lifecycle.

What changed

File Change
src/fi/alk/harness/scenario.py NegativeControl (name, why, steps, must_fail) and ScenarioSource (kinds rule, file, chat_message, evidence_case, requirement, assumption, with the quoted words) on Scenario.
src/fi/alk/harness/prove.py Plays each negative control like a solution; a control fails when a check it names fails. New blocks no_negative_control, negative_control_passed, answer_leaked, source_unresolved; control_not_run when a control's tool has no endpoint here. _outcome reads a refusal case as no_change only when the reference solution really left the world untouched.
src/fi/alk/harness/leaks.py (new) Leaked-answer check: an 8-word run of grader-only text (sub-goal what, judged, reference, verifier, comparison, tests, a control's why, evaluator-only files), a multi-word sub-goal name, or a computed expected value the world does not hold, found in the instruction, variables, persona, opening message or an actor's brief. Returns the excerpt and the grader field.
src/fi/alk/harness/grounding.py (new) Resolves rule, file, requirement and assumption sources against the contract and the world's files (matched_digest as sha256:<hex>); other kinds are left for the platform.
src/fi/alk/harness/prove_suite.py (new) Re-proves every scenario of a suite, writing each proof as it is made; a crash on one case is broken_check, a run out of budget writes the rest validation_incomplete. Keeps a 300 s margin before the job's deadline.
src/fi/alk/harness/authoring_entrypoint.py --certify-only --prove-scenarios [--deadline-at] runs the suite re-prove before sealing and writes scenario-proofs.json; a failure there never fails the certify.
src/fi/alk/harness/scenario_tools.py accept_scenario refuses a case with one of the four codes above, with the reason; controls are checked against the agent's contract like solution steps.
src/fi/alk/harness/proofs.py Format 2 entries also carry outcome, the controls that ran, the leaks found and the source results.
skills/write-scenarios/SKILL.md, references/refusals.md, skills/builder/SKILL.md How to write a task-specific negative control, cite sources, and keep grader-only material out of what the caller and agent see.
tests/harness/*, tests/test_harness.py New and adjusted tests (below). Existing builders that submit through accept_scenario gained a negative control.

Behavior callouts

  • Every new scenario must carry at least one negative control; without one it is not kept (no_negative_control). Existing saved scenarios are not rewritten; the platform reads them as legacy.
  • A scenario with no sources is kept and shown as "assumed by the builder". Only a cited source that does not resolve blocks it.
  • A refusal (no-change) case may hold on an untouched world; its negative control performs the forbidden change, and the checks must fail it.
  • Controls whose tools only run inside the agent read control_not_run: the case is kept as written, never proved.
  • answer_leaked wins over every lane code; a lane code never hides a fault (test_lane_codes_never_hide_a_fault).
  • New CLI flags on the authoring entrypoint (--prove-scenarios, --deadline-at). No new env var.

Tests

  • 25 new tests across test_negative_controls.py, test_leaks.py, test_grounding.py, test_prove_suite.py, test_proofs.py, plus adjusted fixtures in test_nudge_turn.py, test_builder_chat.py and tests/test_harness.py.
  • Run module by module: the touched modules plus every tests/harness module importing changed code (35 modules) and tests/test_harness.py. All new and named tests pass. Failures identical on dev (0203bdf5): test_harness.py 3, test_background_noise.py 2, test_grade.py 1, test_harness_architecture.py 11.
  • Mutation checks (this PR and its base): 32 of 32 killed; 6 weak mutations were redone and test_failure_message_is_not_grader_only strengthened.
  • Break-it pass, throwaway tests: deferred plus leaked, assumed plus a control that passed, an unresolved source, format 1 counts, a crash on the first case, a re-recorded proof, a control naming a check that does not exist, an assumed write outcome, the why text. Two real bugs came out of it and are fixed here: a compared literal inside return None if ... == 42.50 else ... was not read by the leak check, and an assumed write step read no_change, so a write case looked like a refusal case.

Live proof

Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress), builder chat on real demo agents.

Case Result Proof
Describe a case; the builder submits without a control "Not kept. no negative control" three times, until it wrote a task-specific control (a $70 refund instead of the full $140) Image 1
Ask for a case whose instruction repeats a sub-goal's words Refused: "what the caller or the agent sees gives the answer away: instruction repeats ... from sub_goal:identifies_caller_before_prescription.what" Image 2
A rule the case cites is removed Re-prove reads Draft: "no rule of the agent contains the quoted words" Image 3
From a policy file / from CSV records Each case cites the file line or its row Images 4, 5
A refusal case (refund outside the 30-day window) Outcome no change; its forbidden refund is the control (Not run on agent-bound tools) Image 6
A negative control that actually runs Not shown live: no local environment has tools the prove lane can execute; unit tests test_two_refunds_control_fails, test_control_that_passes_blocks Not shown

Builder chat: not kept, no negative control

Leaked answer refused with its reason

Source no longer resolves after a rule change

A case citing a policy file line

A case citing a CSV row

A refusal case

Playwright e2e

None in this repo. The platform flows for the lifecycle (SIM-E2E-008 and SIM-E2E-001 to 006) passed 7 of 7 on the platform branch at 4fb67ad21, with this branch mounted.

Plan lines S3 to S21 and the test pass on this head

Line Change
S3 judged-only A case only a judge grades is proved only when the judge fails a control's wrong transcript; otherwise it is kept as written (ad37f0c5).
S4 live controls For service-bound tools, the reference solution, an empty run and each negative control are played on the live runtime (live_proof.py, d751967e); a check that raises under a control reads broken and a control that cannot be played is no evidence (70e8de2f).
S6 per-case prove_suite --keys proves only the named cases (445444e0).
S11 sources The builder's submit refuses a new or changed case that cites nothing and asks for an assumption source (ec69f384); a blank assumption source does not count (2a29511b).
S16 evidence Each check's proof records what it reads, tool calls or world state (6d08f42c).
S17 policy conflict Each cited rule is judged against the expected outcome; a contradiction blocks the case, an undecided judge never does (e61ebba3).
S19 must-pass A scenario carries must_pass, versioned with its content (f76c3f2b, 23010c01).
S21 grader-only The reference solution's argument values are compared as grader-only material (bf2c058a); 37.5 and 37.50, and values nested in a list or object, count as a leaked answer (d45eba32).

Fixed during the test pass:

Commit Fix
d333f0c3 Found live: a builder save on an older suite showed "1 added, 12 changed" because every rewritten scenario.json gained "must_pass": false. must_pass is written only when set, so an older suite rewritten as it was reads unchanged.
3b2fbf9c The builder submit's require_sources=True had no test (a mutation survived); a test now pins it.
e4481805 Ten fixtures submitted builder cases with no source; they now carry an assumption source as the builder writes them. The gate is unchanged.

Tests on this head (the targeted harness modules, one process, the CI extras installed): 2620 pass. Three failures, the same on dev (0203bdf5): test_hosted_claude_session_uses_durable_resume_and_restricted_tools, test_a_skill_only_names_tools_its_stage_actually_has, test_the_system_prompt_stays_within_budget. With the two branches stacked on it: 2658 pass, the same three.

Mutation checks in this pass, all killed: the accept-time sources gate (2 tests fail), the builder submit sources flag (1), must_pass written only when set (1).

Break-it attempts: a blank or whitespace assumption source (refused), an unchanged legacy case with no sources (saves), a changed legacy case with no sources (refused), a judge that cannot decide a policy conflict (not cached, never blocks), a control naming a check the case does not have (refused with the reason).

Live (production-spec local stack, pharmacy voice demo): asked the builder for a case about a prescription number that is not on file; it wrote the case as "not on file" and cited the person's words. Image on the platform PR #4107.

Scope out

  • Proving in-process (agent-bound) tools by driving the agent: later, it needs the tool wire format; such cases stay "kept as written". Service-bound tools are proved on the live runtime.
  • CLI and SDK parity (validate, archive, sources): later, after the platform contract is released.
  • Re-certifying an environment built by an older harness can fail in the sandbox (source_model_fingerprint_mismatch, generated_runtime_entrypoint_ambiguous), in code this PR does not touch.

#324: fix(harness): reseal the scenarios that load and name the ones that could not be applied

Why

TH-8929: when a builder turn changed several scenarios and one of them could not be loaded the way a run loads it, the reseal failed and the whole turn failed with it, so every good change was lost. Now the good changes save, the case that could not be applied keeps its last sealed version (or stays out if it was never sealed), and the platform is told which case it was and why.

What changed

File Change
src/fi/alk/harness/bundle_author_v2.py scenario_seal_problems(authoring) loads each scenario folder alone, as a run does (scenario_source._load_one), and returns {key: first line of why}. reseal_scenarios(authoring, skip=...) reseals without the failing keys; _copy_scenarios keeps a skipped case's last sealed copy, never under a folder name a case written in the same save uses.
src/fi/alk/harness/hosted_chat_entrypoint.py _save_changes checks first, reseals with the failing keys skipped and publishes with failed_scenarios; the reply adds "Saved, except N scenarios that could not be applied: key: reason." A failure of the whole reseal (contract or seed) no longer fails the turn: the change is kept as the chat's unsaved draft and the reply says so.
src/fi/alk/harness/hosted_conversation.py ConversationClient.publish(digest, failed_scenarios=...) sends {"digest", "failed_scenarios": [{"scenario_key", "reason"}]}, at most the platform's 200, and only when there is one.

Behavior callouts

  • Nothing changes for a turn where every scenario loads.
  • A platform that predates future-agi fix/TH-8929-partial-apply refuses the unknown field; the field is sent only when a case failed, so ordinary saves are unaffected.

Tests

Test What it pins
test_builder_chat.py::test_one_broken_scenario_does_not_block_the_save Reseal skips the failing key, publish names it, the reply says saved except one
::test_a_folder_a_run_cannot_load_keeps_its_sealed_version The failing case keeps its last sealed copy
::test_a_held_copy_never_takes_the_folder_of_a_case_written_now A held copy whose folder name a new case now uses is left out instead of breaking the reseal
::test_a_failed_reseal_keeps_the_change_as_a_draft A whole-reseal failure keeps the change as a draft and the turn completes
::test_publish_sends_at_most_the_platform_limit_of_failed_scenarios 250 failures send 200; no failures send no field

Run on this branch (the targeted harness modules, one process): 2647 pass; three failures, the same on dev (0203bdf5). Mutations: reseal without skip, and the held-copy folder guard, each fail their test.

Break-it attempts: every case failing (each named), a never-sealed failing case (left out, the rest sealed), a held copy whose folder a good case took (fixed here: it raised FileExistsError and failed the whole reseal), 250 failures (capped at 200), a multi-line reason (first line only).

Live proof

The platform half is shown live on future-agi fix/TH-8929-partial-apply (one case of twelve reads "Could not apply" with its reason, the others keep their state, and discard clears it). A builder turn that writes an unloadable folder is not reproducible on purpose: the builder's own gate refuses a broken case before it is written, so the reseal path is shown by the tests above.

#325: feat(harness): record why a rebuild drops a scenario for a removed capability and restore it under its key when the capability returns

Why

TH-8754: when an agent version loses a tool, the scenario rebuild drops the cases that need it, and until now nothing remembered why. When the tool came back, the cases were gone for good: a rebuild wrote new ones under new keys, so their run history no longer lined up.

Now a rebuild records each case it drops for a removed capability (which tools and sub-goals it needed, which were missing, the agent version, a readable reason and the case's last content). When a later rebuild has every capability back, the case comes back under its own key from its last content, is proved like any other case, and is listed as restored.

What changed

File Change
src/fi/alk/harness/dropped.py (new) dropped.json beside the scenarios: record_drops (from the turn-start snapshot, only drops whose needed tools or sub-goals are missing), restore (every record whose capabilities are all back goes into the kept list under its key), finish_restore (a restored case that fails its proof or is dropped again stays recorded with restore_failed and the why), submit_refusal (a rebuild may bring a dropped key back only by restore), restore_note. save({}) removes the file.
src/fi/alk/harness/nudge.py The suite rebuild records drops in report_update, restores at the start of the turn, and reports restored and restore_failed.
src/fi/alk/harness/hosted_chat_entrypoint.py The rebuild's message names the restored cases and any step whose arguments the tool no longer takes.
src/fi/alk/harness/skills/builder/topics/agent-update.md The builder is told not to rewrite a dropped case under a new key, and that a tool the agent's source defines again is added back with declare_tool.
src/fi/alk/harness/amend.py fix_tool and adopt_tool on a tool the contract lacks said "is not a tool this agent has"; they now say it is not in the contract and point at declare_tool. Found live: the builder took the old message at its word and never added the tool back, so nothing could be restored.

Behavior callouts

  • Drop and restore run only in the scenario rebuild that follows an agent change (the suite stage of a rebuild nudge). A person's edit in the builder chat that removes a case records nothing.
  • The platform raises rebuild nudges only with HARNESS_VERSIONING_NUDGES_ENABLED on. Production values do not set it today, so this feature does nothing in production until that flag is on.
  • An ordinary drop (not for a missing capability) is not recorded.

Tests

tests/harness/test_dropped_scenarios.py, 13 tests: a v5 drop records capabilities, agent and reason; an ordinary drop is not recorded; no restore while the capability is still missing; restore under the same keys and last content when it returns; restored cases are proved; restore only once every capability is back; a repeated report_update keeps the restore listed; the message names the restored cases and a changed argument; a restore that cannot be proved stays dropped with why; a restored case dropped again is a failed restore; a rebuild cannot write a new case under a dropped key; editing a restored case is allowed; with no drops left dropped.json is removed and the scenario digest never reads it (so the platform hashes such a suite exactly as before).

Also tests/test_harness.py::test_fixing_a_tool_the_contract_lacks_points_at_declare_tool.

Run on this branch (the targeted harness modules, one process): 2654 pass; three failures, the same on dev (0203bdf5). Mutations: keeping an empty dropped.json fails the last test; the old refusal text fails the declare_tool test.

Break-it attempts (throwaway tests), all hold: a corrupt dropped.json reads as no drops and restores nothing; a record whose content no longer reads as a scenario is refused with why while the others restore; a restore whose name a kept case now holds is refused; restoring twice restores nothing the second time; submit_refusal with empty, null or padded keys.

Live proof

See future-agi feat/TH-8754-restore-dropped-scenarios for the screenshots; the run is summarised here.

Local stack with production sandbox specs, a chat demo environment (deals desk) on its environment v12, the last version before tools were added through chat. Rebuild nudges turned on locally (see the callout). Agent v3 is v1's source with the apply_discount tool removed.

Step Result Proof
Agent v3 becomes the default The card offers a rebuild: "Agent v3 changed ... Rebuild the environment?" Image 1
Rebuild (environment, then scenarios) apply_discount leaves the contract; the suite rebuild drops live-manager-discount-negotiation and manager-initiated-post-discussion-workflow; suite v10 keeps each drop with "agent v3 no longer has apply_discount; the environment retired applied_discount_to_deal", the missing tools and sub-goals, and the agent version; the version diff carries the same reason Image 2, API below
A person adds a new case under a dropped key 422 scenario_key_retired: "live-manager-discount-negotiation was removed earlier and can't be reused; give the scenario a new name" API
Agent v1 (the tool is back) becomes the default; rebuild The environment stage adds apply_discount back with declare_tool (environment v18); the suite rebuild restores both cases under the same keys from their last content, proved by the gates; the card reads "2 scenarios restored" and lists them under Restored Image 3
The restored case Same number (#4) and key; its revisions run from suite v1 to v9 and on to v11, so its history stays linked Image 4

Suite v10's diff (GET versions/suite/{id}/diff/), removed entries:

{"key": "live-manager-discount-negotiation",
 "reason": "agent v3 no longer has `apply_discount`; the environment retired `applied_discount_to_deal`",
 "capabilities": {"tools": ["apply_discount"], "sub_goals": ["applied_discount_to_deal"]},
 "agent_version": "v3"}

1: rebuild offered

2: drop recorded

3: restored

4: restored case keeps its key and history

Found while running it, fixed here and in the harness PR: the rebuild card's change list showed the generic "removed when the environment was updated" instead of the stored reason (ad6acfdcf); and on the first attempt the environment rebuild did not put apply_discount back (the harness told the builder it was "not a tool this agent has" when it was only missing from the contract), so nothing was restored. With the harness fix the second attempt restored both cases.

The restored cases read Draft "No negative control" on the Scenarios tab: they were written before negative controls existed, which the lifecycle reports for any such case.

Linear

Linear: TH-8752
Linear: TH-8918
Linear: TH-8929
Linear: TH-8754

🤖 Generated with Claude Code

https://claude.ai/code/session_019gWFtEXoqSfwg3LAdyoEUb

@KarthikAvinashFI KarthikAvinashFI self-assigned this Nov 14, 2026
… a control's wrong transcript, else kept as written
…d named on publish, and a failed reseal keeps the change as a draft
…pability, and restore it under its key once the capability returns
… publish so a larger set never refuses the save
…en, and a control that cannot be played is no evidence
… nested in a list or object, counts as a leaked answer
… declare_tool; fix_tool and adopt_tool say so instead of claiming the agent lacks it
@KarthikAvinashFI KarthikAvinashFI changed the title feat(harness): negative controls, leaked-answer check, grounding and suite proving (TH-8918) feat(harness): scenario proof honesty and lifecycle (TH-8752, TH-8918, TH-8929, TH-8754) Nov 15, 2026
@KarthikAvinashFI
KarthikAvinashFI changed the base branch from feat/TH-8752-proof-honesty to dev November 15, 2026 19:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant