Skip to content

fix(harness): restore ask routing on the Claude backend and repair stale tests on dev - #186

Closed
KarthikAvinashFI wants to merge 15 commits into
devfrom
fix/alk-dev-red-tests
Closed

KarthikAvinashFI wants to merge 15 commits into
devfrom
fix/alk-dev-red-tests

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

1. Why

Run file by file, dev has 39 failing tests in 11 files, plus one test that does not finish in ten minutes. Most failures are tests left behind by intentional changes, and one is a code regression in the Claude backend. This PR fixes 23 of them; the rest need a decision from the owners of that code and are listed in section 7 with their cause.

2. What changed

File Failing on dev Cause, and the commit that introduced it Fix
src/fi/alk/harness/backends/claude.py test_hosted_claude_session_uses_durable_resume_and_restricted_tools A 23 Sep refactor of the backend (36158431) dropped two lines from the hosted session options: the ask tool left out of allowed_tools, and tools limited to the session's built-ins. An allowed tool is approved before the permission callback runs, so a session's ask callback is never called and a question cannot reach the person Restores both: the ask tool goes through the permission callback, and tools is the built-ins of the session and its workers (workers draw theirs from the session's set) plus the delegation tools when it has workers
src/fi/alk/harness/hosted_chat_entrypoint.py none (found while checking the change above) With the ask tool going through the permission callback, a question during a platform build turn now reaches _ask, which returned a tool result rather than a permission decision During platform turns on the Claude backend it returns a deny carrying the same "decide it yourself" message
tests/test_harness.py same test The test also used a Claude model, which the harness now refuses (36158431) Uses vertex_ai/gemini-3.7-flash through the gateway, as the routing test next to it does
tests/harness/test_dispatch_ack.py 4 The call runner now reads failure.details, failure.stage, report.run_id, case.test_case_id and the runner's usage context (bbe66df2, 6226469d); the fakes predate that The fakes use the real SimulationFailure model and carry those fields
tests/harness/test_grade.py test_the_cap_holds_for_the_session_that_saves_too (NameError: st) A 23 Sep merge (057e9c61) left the body of a removed generate_suite test inside this test; that tool was removed in 36158431 Drops the leftover body; the cap test itself is unchanged
src/fi/alk/harness/skills/write-scenarios/SKILL.md, tests/test_harness.py test_a_skill_only_names_tools_its_stage_actually_has The skill names example tools from imagined agents (create_booking, verify_code, ...) and two words a check example reads, and calls the fixture fields setup/ready The skill names the fields as the model has them (setup_code/ready_code); the example names join the test's existing list of example names. After dev was merged in, the skills also name the agent_reader sub-agent, the both verifier value and the caller_phone fixture key; the test reads those from where they are defined
tests/test_harness.py test_the_system_prompt_stays_within_budget (151,205 vs 90,000 chars) The writer skills grew through deliberate rule changes from 23 Sep to 21 Oct; the cap was set before them. No duplicated or dead text was found (under 500 duplicated characters) The caps are set from the skills as they stand plus about a tenth: 165,000 and 245,000
tests/test_simulator_voice_instructions.py test_the_caller_asks_again_for_the_part_of_its_question_left_unanswered Rule 12e was reworded on 1 Oct (525c5df4); the rule is still there The test checks the current wording
tests/test_harness_architecture.py 11 Hosted jobs now require a pinned commit, a dedicated VM and platform secret references (beface8c, 25 Aug), and a GitHub checkout verifies the pinned commit with git rev-parse The fixtures carry those fields; the checkout fakes answer rev-parse
src/fi/alk/harness/scenario.py, prove.py, DESIGN.md test_no_competitor_names_in_shipped_tree Three sentences named another benchmark Reworded without the name
tests/test_phase7_persona_studio.py 2 (HTTP 401) Importing the config writes the default API URL into FI_BASE_URL, which the client reads before the AGENT_LEARNING_API_URL the fixture set, so the test called the real API instead of its local stub The fixture sets all three URL variables, so the pull stays on localhost

3. Behavior callouts

  • Claude backend: a session's AskUserQuestion now goes through its ask callback again. A session with no ask callback is refused the tool with the gate's usual message instead of having it approved. Built-in tools are limited to those the session and its workers grant, plus Agent and Task when it has workers; the permission hook already denied any other tool, so this changes what is offered, not what may run. During a platform build or check turn, the CLI's question tool is denied with the usual "decide it yourself" message instead of returning a tool result. This changes hosted builder and authoring sessions on this backend; the backend's unit tests pass, but it has not been run in a live build.

4. Tests

Each file below was run on its own with this branch's src on the path.

File dev This branch
tests/harness/test_dispatch_ack.py 4 failed 17 passed
tests/harness/test_grade.py 1 failed 23 passed
tests/test_harness.py 3 failed 339 passed
tests/test_simulator_voice_instructions.py 1 failed 6 passed
tests/test_harness_architecture.py 11 failed 43 passed
tests/test_no_competitor_names.py 1 failed 1 passed
tests/test_phase7_persona_studio.py 2 failed 33 passed
tests/harness/test_claude_workers.py, test_builder_chat.py (21, incl. a new platform-turn case), test_vertex_gemini_backend.py, test_chat_worker.py, test_chat_call_runner.py, test_retell_chat_call_runner.py passed passed

Every other test file in tests/ passed on this branch, run file by file, except those in section 7. After the latest dev was merged in, the whole suite was rerun file by file on this branch and on dev: this branch has no failure that dev does not also have, and the remaining ones are those in section 7.

5. Commands to run the tests

uv run pytest -q tests/harness/test_dispatch_ack.py tests/harness/test_grade.py tests/test_harness.py \
  tests/test_simulator_voice_instructions.py tests/test_harness_architecture.py \
  tests/test_no_competitor_names.py tests/test_phase7_persona_studio.py

Terminal output

<image: pytest output for the files above>

7. Scope in / scope out

Not fixed here, each needing a decision from the owners of that code:

  • tests/test_cli_examples.py (11) and tests/test_init_golden_paths.py (2): the shipped optimization and red-team examples fall below their 0.95 thresholds since c239aceb (6 Aug) stopped counting unconfigured metrics as 1.0. The file passes on the commit before it and fails on it. In the example inspected, the best candidate scores 0.9353 because its task is not completed (task completion 0.52). Lowering the thresholds would hide that; the examples or their thresholds need their owners' call.
  • tests/test_harness_sandbox_server.py (2): the sandbox server builds hosted jobs without a dedicated-VM runtime or a pinned commit, which hosted jobs require since beface8c, so a hosted GitHub submission is refused. Whether the server should set dedicated-VM isolation itself, or require it from the caller, is a design question.
  • tests/test_harness_secrets.py::test_worker_never_inherits_host_voice_provider_credentials: the worker environment passes CARTESIA_API_KEY through since f0b3fcea (27 Aug), on purpose, so hosted callers get distinct voices; the test asserts it never does. Which one is right depends on whether hosted speech now goes only through the gateway.
  • tests/test_config_and_facades.py: test_agent_learn_release_check_reports_v1_milestones runs every shipped optimization example and did not finish within ten minutes here; its pass condition likely depends on the same examples as above, which was not confirmed.

Linear

None

@KarthikAvinashFI KarthikAvinashFI self-assigned this Oct 22, 2026
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Superseded by #323.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant