test(eval): pin the orchestrator-executor evaluator entry contract - OrchestratorExecutorEvaluator (Closes #1804)

[dmitrii-f-t27]

Closes #1804

What changed

One new file: trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (the path the issue's Success Criteria names). The subject orchestrator-executor/index.ts is untouched.

The single exported class is exercised (4 it( blocks) on its no-environment entry validation - every path measured to refuse BEFORE any CDP connection, after the user query has been logged (the recording fake captures the logged query):

• A config whose agent type is not orchestrator-executor answers the pinned refusal.
• An orchestrator without a model answers orchestrator.model is required in config.
• An executor without a model answers executor.model is required in config.
• A clado-action executor without a baseUrl answers executor.baseUrl is required in config for clado-action provider.

Not pinned, named in the suite header: the full execute path (live browser via CDP, agent backends, capture flow) - the entry validation contract above is what can be pinned without any environment.

The capture context is a recording fake; the suite registers no mocks. No network, no database, no container.

How it was verified (FR-004)

• Passing run against the unmodified subject: 4 pass / 0 fail / 12 expect() calls, Ran 4 tests across 1 file.
• Deliberately broken copy (the type check inverted; the orchestrator-model check gutted; the clado error reworded; the query logging dropped): 4 fail out of 4, quoted raw above. The subject was restored byte-identical (git checkout) and the suite passes again.
• All error texts were measured on the live module first (one probe round). The gate caught one typecheck error before pushing (AgentContext lives in ../types, not ../../types) - fixed; after it: biome ci 0 Found, typecheck base set with 0 mentions of the new file; the one-process run of the sibling eval suites (agents/utils/runner, 28 tests) is green.

Not verified

• Nothing beyond the quoted local runs; the CI Tests / eval job re-runs the suite on this SHA.

Dmitrii Fedorov (@dmitrii-f-t27)

[tri-claw]

Summary by t27-bees

This PR adds a test file that pins the entry validation contract of OrchestratorExecutorEvaluator: four tests verify the evaluator refuses invalid configs (wrong agent type, missing orchestrator model, missing executor model, missing clado-action baseUrl) before any CDP connection, after logging the user query.

Worth a look

• trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (lines 38–58): The fakeContext helper casts a partial object to AgentContext via as unknown as AgentContext. If AgentContext requires properties not provided here (e.g., capture.screenshot is typed as never, getErrors returns unknown[]), the tests could pass while the real evaluator fails at runtime when accessing those missing members.
• trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (line 48): screenshot: {} as never makes screenshot uncallable/inspectable; if the evaluator under test accesses capture.screenshot in the validation path, it will throw at runtime despite tests passing.
• trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (line 50): getErrors: () => [] as unknown[] returns unknown[]; if the evaluator reads errors during validation and expects a specific shape, the test may not exercise that code path.
• trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (lines 60–68): expectRefusal catches any thrown value but only asserts it is an Error with the exact message. If the evaluator throws a non-Error (e.g., a string) or a subclass with extra properties, the test would still pass the message check but the real error handling could differ.
• trios/agent-server/apps/eval/src/agents/orchestrator-executor/index.test.ts (line 70): ORCHESTRATOR constant uses model: 'gpt-test'; if the validation logic changes to also require a non-empty string or specific format, this test constant would need updating, but the diff doesn't show the implementation to confirm current requirements.

---

_{Read: 1 files, +123 -0, the whole diff. Head 4ca668f. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.}

[github-actions]

✅ Tests passed — 3562/3632

Suite	Passed	Failed	Skipped
✅ agent	190/190	0	0
✅ build	9/9	0	0
✅ cdp-protocol	5/5	0	0
✅ eval	93/93	0	0
✅ server-agent	280/280	0	0
✅ server-api	2159/2228	0	69
✅ server-browser	6/6	0	0
✅ server-integration	10/11	0	1
✅ server-lib	279/279	0	0
✅ server-pglive	166/166	0	0
✅ server-root	68/68	0	0
✅ server-sim	8/8	0	0
✅ server-skills	31/31	0	0
✅ server-tools	244/244	0	0
✅ shared	14/14	0	0

View workflow run

[dmitrii-f-t27]

Checks note, before merge:

• runner / Biome and runner / Typecheck are red on the base branch queen with pre-existing failures (reference base run https://github.com/gHashTag/trios/actions/runs/38087839871 on dfb6d97). On this PR's SHA: the Typecheck log has the 19 base error TS lines and 0 mentions of the new orchestrator-executor/index.test.ts; the Biome log has 0 diagnostics for the new file. The suite itself passed in Tests / eval on this SHA (4 tests), and all other Tests / * suites including server-pglive are green.

Dmitrii Fedorov (@dmitrii-f-t27)