Repository navigation
fix(harness): restore ask routing on the Claude backend and repair stale tests on dev - #186
Closed
KarthikAvinashFI wants to merge 15 commits into
Closed
KarthikAvinashFI wants to merge 15 commits into
KarthikAvinashFI wants to merge 15 commits into
Conversation
…nner fields the call runner now reads
…restore the session's built-in tool list on the Claude backend
… and list the skill's example tool names
…and secret references hosted jobs require
…eads first, so the pull never leaves localhost
…latform turns and offer workers their own built-ins
…r names from where they are defined
This was referenced Oct 23, 2026
…into fix/alk-dev-red-tests
Merged
5 of 10 tasks
Member
Author
|
Superseded by #323. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Why
Run file by file,
devhas 39 failing tests in 11 files, plus one test that does not finish in ten minutes. Most failures are tests left behind by intentional changes, and one is a code regression in the Claude backend. This PR fixes 23 of them; the rest need a decision from the owners of that code and are listed in section 7 with their cause.2. What changed
devsrc/fi/alk/harness/backends/claude.pytest_hosted_claude_session_uses_durable_resume_and_restricted_tools36158431) dropped two lines from the hosted session options: the ask tool left out ofallowed_tools, andtoolslimited to the session's built-ins. An allowed tool is approved before the permission callback runs, so a session'saskcallback is never called and a question cannot reach the persontoolsis the built-ins of the session and its workers (workers draw theirs from the session's set) plus the delegation tools when it has workerssrc/fi/alk/harness/hosted_chat_entrypoint.py_ask, which returned a tool result rather than a permission decisiontests/test_harness.py36158431)vertex_ai/gemini-3.7-flashthrough the gateway, as the routing test next to it doestests/harness/test_dispatch_ack.pyfailure.details,failure.stage,report.run_id,case.test_case_idand the runner's usage context (bbe66df2,6226469d); the fakes predate thatSimulationFailuremodel and carry those fieldstests/harness/test_grade.pytest_the_cap_holds_for_the_session_that_saves_too(NameError: st)057e9c61) left the body of a removedgenerate_suitetest inside this test; that tool was removed in36158431src/fi/alk/harness/skills/write-scenarios/SKILL.md,tests/test_harness.pytest_a_skill_only_names_tools_its_stage_actually_hascreate_booking,verify_code, ...) and two words a check example reads, and calls the fixture fieldssetup/readysetup_code/ready_code); the example names join the test's existing list of example names. Afterdevwas merged in, the skills also name theagent_readersub-agent, thebothverifier value and thecaller_phonefixture key; the test reads those from where they are definedtests/test_harness.pytest_the_system_prompt_stays_within_budget(151,205 vs 90,000 chars)tests/test_simulator_voice_instructions.pytest_the_caller_asks_again_for_the_part_of_its_question_left_unanswered525c5df4); the rule is still theretests/test_harness_architecture.pybeface8c, 25 Aug), and a GitHub checkout verifies the pinned commit withgit rev-parserev-parsesrc/fi/alk/harness/scenario.py,prove.py,DESIGN.mdtest_no_competitor_names_in_shipped_treetests/test_phase7_persona_studio.pyHTTP 401)FI_BASE_URL, which the client reads before theAGENT_LEARNING_API_URLthe fixture set, so the test called the real API instead of its local stub3. Behavior callouts
AskUserQuestionnow goes through itsaskcallback again. A session with noaskcallback is refused the tool with the gate's usual message instead of having it approved. Built-in tools are limited to those the session and its workers grant, plusAgentandTaskwhen it has workers; the permission hook already denied any other tool, so this changes what is offered, not what may run. During a platform build or check turn, the CLI's question tool is denied with the usual "decide it yourself" message instead of returning a tool result. This changes hosted builder and authoring sessions on this backend; the backend's unit tests pass, but it has not been run in a live build.4. Tests
Each file below was run on its own with this branch's
srcon the path.devtests/harness/test_dispatch_ack.pytests/harness/test_grade.pytests/test_harness.pytests/test_simulator_voice_instructions.pytests/test_harness_architecture.pytests/test_no_competitor_names.pytests/test_phase7_persona_studio.pytests/harness/test_claude_workers.py,test_builder_chat.py(21, incl. a new platform-turn case),test_vertex_gemini_backend.py,test_chat_worker.py,test_chat_call_runner.py,test_retell_chat_call_runner.pyEvery other test file in
tests/passed on this branch, run file by file, except those in section 7. After the latestdevwas merged in, the whole suite was rerun file by file on this branch and ondev: this branch has no failure thatdevdoes not also have, and the remaining ones are those in section 7.5. Commands to run the tests
Terminal output
<image: pytest output for the files above>
7. Scope in / scope out
Not fixed here, each needing a decision from the owners of that code:
tests/test_cli_examples.py(11) andtests/test_init_golden_paths.py(2): the shipped optimization and red-team examples fall below their 0.95 thresholds sincec239aceb(6 Aug) stopped counting unconfigured metrics as 1.0. The file passes on the commit before it and fails on it. In the example inspected, the best candidate scores 0.9353 because its task is not completed (task completion 0.52). Lowering the thresholds would hide that; the examples or their thresholds need their owners' call.tests/test_harness_sandbox_server.py(2): the sandbox server builds hosted jobs without a dedicated-VM runtime or a pinned commit, which hosted jobs require sincebeface8c, so a hosted GitHub submission is refused. Whether the server should set dedicated-VM isolation itself, or require it from the caller, is a design question.tests/test_harness_secrets.py::test_worker_never_inherits_host_voice_provider_credentials: the worker environment passesCARTESIA_API_KEYthrough sincef0b3fcea(27 Aug), on purpose, so hosted callers get distinct voices; the test asserts it never does. Which one is right depends on whether hosted speech now goes only through the gateway.tests/test_config_and_facades.py:test_agent_learn_release_check_reports_v1_milestonesruns every shipped optimization example and did not finish within ten minutes here; its pass condition likely depends on the same examples as above, which was not confirmed.Linear
None