fix(evals): include agent output in AgentRunHistory [AE-2147]

[rakesh-uipath]

Fixes AE-2147 — reported by James Dickson in #help-agent-builder, filed by Maria Vimer.

The bug

{{AgentRunHistory}} reaches the trajectory judge as a tool-call transcript with the agent's own output missing, so the judge grades an incomplete trace.

It is real, and it is one predicate:

https://github.com/UiPath/uipath-python/blob/30f6bec2/packages/uipath/src/uipath/eval/_helpers/evaluators_helpers.py#L653

for span in workload_trace:
    if span.attributes and (tool_name := span.attributes.get(TOOL_NAME_ATTR)):

A span without tool.name is skipped, and the docstring said so out loud: "Only processes tool spans". The workload's own answer lives on WorkloadExecution.workload_output (populated in runtime.py:1143) and never entered the string.

There is no second channel for it. On BaseLLMTrajectoryEvaluator the actual-output placeholder is the history:

actual_output_placeholder: str = "{{AgentRunHistory}}"   # llm_judge_trajectory_evaluator.py:63
...
return trace_to_str(workload_execution.workload_trace)   # line 83 — output not passed

So the judge never saw the output at all — while its own system prompt promises it "a trace/history of the agent's actions and outputs".

Reproduced against James's run (one Web_Search tool span, agent answered correctly). Before:

[2026-08-26 18:46:40] LLM Response:
  Agent Selected 1 Tool(s):

  Tool: Web_Search
  Arguments: {}

[2026-08-26 18:46:47] Tool Call Response - Web_Search:
{}

Byte-for-byte what his screenshot shows. The judge scored it 10% with the justification "It never produced a user-facing answer identifying Argentina as the most recent World Cup winner" — while the Actual Output panel on the same screen read {"search_results_answer": "Argentina won the most recent FIFA World Cup (Qatar 2022)..."}. The agent was right and got marked down for the serialiser's omission.

The fix

trace_to_str takes the workload output and appends it as a final block; both trajectory evaluators pass it.

[2026-08-26 18:46:47] Tool Call Response - Web_Search:
{}

Agent Output:
{"search_results_answer": "Argentina won the most recent FIFA World Cup (Qatar 2022), ..."}

Fixed in LLMJudgeTrajectoryEvaluator / LLMJudgeTrajectorySimulationEvaluator and in LegacyTrajectoryEvaluator, which had the same defect through the same helper.

Deliberately not changed

Intermediate assistant messages are still not rendered. LLM spans carry the full system prompt, and test_legacy_trajectory_prompt_uses_compact_tool_history exists precisely to keep a 10k-char prompt out of the judge's context. Surfacing intermediate reasoning needs a bounded projection of the LLM spans, not a filter removal — worth doing, separate change. James's other asks in that thread (documenting what each {{...}} binds to in the evaluator UI, eval-driven judge authoring) are product work, not this.

workload_output=None keeps the old behaviour, so no existing caller changes shape.

Tests

packages/uipath/tests/evaluators/test_trajectory_agent_output.py — reproduces the AE-2147 run through both evaluators and asserts the answer reaches the prompt. Both fail on main:

E  AssertionError: assert 'Argentina won the most recent FIFA World Cup (Qatar 2022).' in
   '...Tool Call Response - Web_Search:\n{}\n'

Plus five cases on trace_to_str itself in test_evaluator_helpers.py (ordering, str vs dict output, omitted output, empty output).

Verification

Ran locally, all from packages/uipath:

• uv run pytest — green, exit 0
• uv run ruff check . — All checks passed
• uv run ruff format --check . — 335 files already formatted
• uv run mypy --config-file pyproject.toml . — no issues in 335 source files
• uv run python scripts/lint_httpx_client.py — clean

Version bumped 2.14.14 → 2.14.15; 2.14.14 is already on PyPI so check-version-availability would have failed otherwise.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The legacy evaluator still omits supplied output for runs with no trace spans.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes {{AgentRunHistory}} so trajectory evaluators include the agent’s final output.

Changes:

• Appends workload output to serialized traces.
• Updates modern and legacy trajectory evaluators.
• Adds regression tests.
• Bumps version to 2.14.15.

File summaries

File	Summary
packages/uipath/uv.lock	Updates locked package metadata.
packages/uipath/tests/evaluators/test_trajectory_agent_output.py	Adds evaluator regression coverage.
packages/uipath/tests/evaluators/test_evaluator_helpers.py	Tests trace output serialization.
packages/uipath/src/uipath/eval/evaluators/llm_judge_trajectory_evaluator.py	Passes agent output into judge history.
packages/uipath/src/uipath/eval/evaluators/legacy_trajectory_evaluator.py	Passes output into legacy history, but empty-trace runs still need correction.
packages/uipath/src/uipath/eval/_helpers/evaluators_helpers.py	Serializes and appends agent output.
packages/uipath/pyproject.toml	Bumps the package version.

Review details

• Files reviewed: 6/7 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Chibionos]

/deep-review — PR #1894 · fix(evals): include agent output in AgentRunHistory [AE-2147]

Branch: fix/ae-2147-trajectory-agent-output → main | Size: +309/−12 across 7 files

Verdict

Approve — correct, minimal, well-tested bugfix; both lenses (architecture + runtime) clean, Python domain pillar clean, pytest/ruff check/ruff format --check/mypy all pass on the changed files.

Slop meter

🎚️ Slop Meter — PR #1894

Verdict: 🟢 clean
Slop axis: 2/100 · Tunnel-vision axis: 0/100
Diff: +309 LOC across 7 files

Only signal: hardcoded_version (2 hits, pyproject.toml + uv.lock) — the legitimate version bump described in the PR body (2.14.14 already published to PyPI). No other slop or tunnel-vision signals fired.

Codex recheck

Codex CLI is on PATH but the exec call failed with "You hit your spend cap set by the owner of your workspace" (not the documented "not on PATH" case). Treated as unavailable for this run — proceeding on local two-lens findings only (Codex not available this run — stronger guarantee once budget/config is fixed).

Findings index (all inline below)

#	Sev	File:Line	Headline
—	—	—	No P0/P1/P2 findings

(No finding bodies in the summary — every body would be an inline comment anchored to a line; there are none to anchor this cycle.)