Skip to content

fix(eval): make task prompts and fixtures read like a real project - #20

Open
manykarim wants to merge 4 commits into
mainfrom
fix/realistic-eval-tasks
Open

manykarim wants to merge 4 commits into
mainfrom
fix/realistic-eval-tasks

Conversation

@manykarim

Copy link
Copy Markdown
Collaborator

Summary

Agents read every file of a fixture, and several of them told the agent it was being evaluated, or gave away the trap of an adversarial task. The PlatynUI and AppiumLibrary prompts sent agents to a committed libdoc spec, so they parsed JSON instead of checking the installed library. That is what led to the misleading robotcodedev/robotcode#689.

Stacked on #19 (its commit is the first one here). Merge #19 first.

What was unrealistic

Where Problem
5 PlatynUI/Appium prompts "a libdoc spec is in specs/": no real project has one, and the library is installed.
sut-failing/README.md Named the bug and said "the shortcuts the task tempts; its graders reject them" (adv-make-it-pass).
sut-legacy-style/README.md Explained the adversarial trap: "the task passes only when the new file uses modern syntax".
sut-selenium/README.md "used by the adversarial 'element count' task".
sut-pylib, sut-language-*, sut-api, sut-minimal, sut-browser READMEs Named the tasks, the hidden grader suites or "the harness".
5 smoke tests "Irrelevant to any eval task".
13 pyproject.toml descriptions "fixture for the rf-agentskills eval harness", "adversarial ... task".
narrow-non-rf-control-01 Told the agent not to use libdoc, rf-mcp or skills, which made the negative control meaningless.
narrow-rf-mcp-execute-01 "they are the subject of this evaluation".

Changes

  • Specs out of the workspace: eval/fixtures/sut-{appium,platynui}/specs/ → eval/graders/specs/ (grader data, never staged, like eval/graders/language/). keywords_resolve resolves a relative spec_dir against the repository; the tasks use spec_dir: eval/graders/specs. The prompts ask agents to check keywords against the installed library. The versions match uv.lock (AppiumLibrary 3.2.1, PlatynUI 0.12.0.dev330).
  • Fixtures read like projects: READMEs, smoke-test docs and descriptions describe the project only. Maintainer notes stay in eval/fixtures/README.md, which isn't staged.
  • Prompts: the two meta prompts above are now worded as a user would write them.
  • Tests:
    • test_fixture_reads_like_a_real_project checks every fixture file for eval wording;
    • test_fixtures_ship_no_libdoc_specs;
    • a spec_dir test.
  • Docs: eval/graders/specs/README.md, the keywords_resolve row in the tasks README, the fixture checklist, and docs/ci/desktop-live-check.md.

Baselines

Every task and fixture hash changes, so the gate reports rebaseline-needed until the GLM 5.3 Flash baselines are re-recorded. A re-baseline run is dispatched on this branch, and its baselines will be committed here.

Testing

  • pytest tests/eval: 665 passed. The 4 Browser tests that fail locally need rfbrowser init; CI runs it.
  • ruff, mypy and validate-tasks (all plugins): clean. check-docs-voice: OK.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP

manykarim and others added 2 commits November 16, 2026 09:26
PlatynUI 0.12.0.dev330 needs Python 3.12+, but the fixture declared
requires-python >=3.10. uv then can't resolve the project, so `uv run`
in an eval workspace fails and agents can't use the fixture's own
environment. Raise the floor to 3.12 and add a test for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Agents read every file in a fixture, and several told them they were in an
eval: smoke tests "irrelevant to any eval task", the sut-failing README
naming the bug and the shortcuts its graders reject, sut-legacy-style
explaining the adversarial trap, sut-selenium naming the "adversarial
element count task". The PlatynUI and AppiumLibrary prompts sent agents to
a committed libdoc spec ("a libdoc spec is in specs/"), so they parsed JSON
instead of checking the installed library as a user would.

- Move the specs to eval/graders/specs (grader data, never staged);
  keywords_resolve resolves a relative spec_dir against the repository.
- The five spec tasks ask agents to check keywords against the installed
  library.
- Fixture READMEs, smoke tests and pyproject descriptions describe the
  project only; maintainer notes stay in eval/fixtures/README.md.
- narrow-non-rf-control-01 no longer tells the agent to avoid RF tooling
  (that made the control meaningless); narrow-rf-mcp-execute-01 no longer
  calls itself an evaluation.
- Tests: fixtures contain no eval wording and no specs; spec_dir works.

Task and fixture hashes change, so the GLM baselines need a re-baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.66
Harness estimate (OpenRouter list prices) 0.26
Credit remaining on the key 2.2048532419999987

Gate

gate: PASS (12 task(s) checked)

rf-skill-eval report

Runs: 36 · pass: 36 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.26 · turn-limit stops: 4

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-language-bdd-01 rf-language narrow treatment 3 100% 100% 0 139654 952 11.0 17.2 0.0045 (est.) no
narrow-language-bdd-01 rf-language narrow baseline (stored) 3 100% 100% 0 78147 1850 7.7 18.7 0.0027 (est.) –
narrow-language-bdd-01 rf-language narrow Δ T−B +0 pp +61507 -897 +3.3 -1.4 +0.0018
narrow-language-data-driven-01 rf-language narrow treatment 3 100% 100% 0 184416 3125 12.7 39.0 0.0061 (est.) no
narrow-language-data-driven-01 rf-language narrow baseline (stored) 3 33% 33% 0 111519 3639 10.0 39.6 0.0040 (est.) –
narrow-language-data-driven-01 rf-language narrow Δ T−B +67 pp +72897 -515 +2.7 -0.6 +0.0021
narrow-language-embedded-01 rf-language narrow treatment 3 100% 100% 0 146175 1882 11.3 23.8 0.0047 (est.) no
narrow-language-embedded-01 rf-language narrow baseline (stored) 3 100% 100% 0 147330 4435 11.3 51.4 0.0052 (est.) –
narrow-language-embedded-01 rf-language narrow Δ T−B +0 pp -1155 -2553 +0.0 -27.7 -0.0005
narrow-language-env-varfiles-01 rf-language narrow treatment 3 100% 100% 0 557399 6899 22.0 (3 at limit) 105.1 0.0180 (est.) no
narrow-language-env-varfiles-01 rf-language narrow baseline (stored) 3 100% 100% 0 163656 2365 15.7 22.0 0.0052 (est.) –
narrow-language-env-varfiles-01 rf-language narrow Δ T−B +0 pp +393743 +4535 +6.3 +83.0 +0.0128
narrow-language-suite-setup-tags-01 rf-language narrow treatment 3 100% 100% 0 279587 3811 19.3 42.9 0.0091 (est.) no
narrow-language-suite-setup-tags-01 rf-language narrow baseline (stored) 3 100% 100% 0 100797 1391 12.3 15.9 0.0034 (est.) –
narrow-language-suite-setup-tags-01 rf-language narrow Δ T−B +0 pp +178790 +2420 +7.0 +27.1 +0.0057
narrow-non-rf-control-01 plugin narrow treatment 3 100% 100% 0 31589 187 2.3 4.1 0.0011 (est.) no
narrow-non-rf-control-01 plugin narrow baseline (stored) 3 100% 100% 0 25754 113 2.0 2.9 0.0009 (est.) –
narrow-non-rf-control-01 plugin narrow Δ T−B +0 pp +5835 +74 +0.3 +1.3 +0.0002
narrow-python-library-flaky-listener-01 rf-python-library narrow treatment 3 100% 100% 0 238744 2571 15.3 34.2 0.0076 (est.) no
narrow-python-library-flaky-listener-01 rf-python-library narrow baseline (stored) 3 100% 100% 0 144494 4530 11.3 47.1 0.0050 (est.) –
narrow-python-library-flaky-listener-01 rf-python-library narrow Δ T−B +0 pp +94250 -1960 +4.0 -12.9 +0.0026
narrow-python-library-inventory-scope-01 rf-python-library narrow treatment 3 100% 100% 0 250575 2900 14.0 42.1 0.0081 (est.) no
narrow-python-library-inventory-scope-01 rf-python-library narrow baseline (stored) 3 67% 67% 0 228079 7098 14.7 (1 at limit) 52.6 0.0079 (est.) –
narrow-python-library-inventory-scope-01 rf-python-library narrow Δ T−B +33 pp +22496 -4198 -0.7 -10.5 +0.0002
narrow-python-library-mode-literal-01 rf-python-library narrow treatment 3 100% 100% 0 303237 3997 16.3 (1 at limit) 57.4 0.0099 (est.) no
narrow-python-library-mode-literal-01 rf-python-library narrow baseline (stored) 3 100% 100% 0 121443 5843 9.3 52.8 0.0047 (est.) –
narrow-python-library-mode-literal-01 rf-python-library narrow Δ T−B +0 pp +181794 -1846 +7.0 +4.6 +0.0052
narrow-rf-injection-positive-01 plugin narrow treatment 3 100% 100% 0 90870 1103 6.3 15.9 0.0031 (est.) no
narrow-rf-injection-positive-01 plugin narrow baseline (stored) 3 100% 100% 0 76938 907 5.7 21.0 0.0025 (est.) –
narrow-rf-injection-positive-01 plugin narrow Δ T−B +0 pp +13932 +196 +0.7 -5.1 +0.0006
narrow-rf-mcp-execute-01 plugin narrow treatment 3 100% 100% 0 262473 1372 12.3 22.2 0.0085 (est.) no
narrow-rf-mcp-execute-01 plugin narrow baseline (stored) 3 100% 100% 0 132129 988 8.7 14.2 0.0046 (est.) –
narrow-rf-mcp-execute-01 plugin narrow Δ T−B +0 pp +130344 +384 +3.7 +8.0 +0.0039
narrow-setup-requests-01 rf-setup narrow treatment 3 100% 100% 0 181447 1344 12.0 19.8 0.0057 (est.) no
narrow-setup-requests-01 rf-setup narrow baseline (stored) 3 100% 100% 0 85587 883 8.0 15.7 0.0028 (est.) –
narrow-setup-requests-01 rf-setup narrow Δ T−B +0 pp +95860 +461 +4.0 +4.1 +0.0029

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-language 5 100% 87% +13 pp +141156 +3.9 +0.0044
rf-python-library 3 100% 89% +11 pp +99513 +3.4 +0.0027
rf-setup 1 100% 100% +0 pp +95860 +4.0 +0.0029

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-language-bdd-01 loaded_rf_language_skill: 3/3
  • narrow-language-data-driven-01 loaded_rf_language_skill: 3/3
  • narrow-language-embedded-01 loaded_rf_language_skill: 3/3
  • narrow-language-env-varfiles-01 loaded_rf_language_skill: 3/3
  • narrow-language-suite-setup-tags-01 loaded_rf_language_skill: 3/3
  • narrow-non-rf-control-01 bounded_tool_calls: 3/3
  • narrow-non-rf-control-01 no_libdoc_invocation: 3/3
  • narrow-non-rf-control-01 no_rf_mcp_calls: 3/3
  • narrow-python-library-flaky-listener-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-inventory-scope-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-mode-literal-01 loaded_rf_python_library_skill: 3/3
  • narrow-rf-injection-positive-01 leveraged_rf_lookup: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_before_write: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_called_at_least_twice: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_had_successful_result: 3/3
  • narrow-setup-requests-01 used_uv_add: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.34
Harness estimate (OpenRouter list prices) 0.06
Credit remaining on the key 2.515285919

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.06

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-requests-users-01 rf-requests narrow treatment 3 100% 100% 0 244197 3117 16.3 40.6 0.0079 (est.) no
narrow-requests-users-01 rf-requests narrow baseline (stored) 3 100% 100% 0 77465 720 9.3 10.7 0.0026 (est.) –
narrow-requests-users-01 rf-requests narrow Δ T−B +0 pp +166732 +2396 +7.0 +29.9 +0.0053
narrow-restinstance-schema-01 rf-restinstance narrow treatment 3 100% 100% 0 348528 4336 20.3 72.1 0.0113 (est.) no
narrow-restinstance-schema-01 rf-restinstance narrow baseline (stored) 3 100% 100% 0 88836 1230 9.0 21.1 0.0029 (est.) –
narrow-restinstance-schema-01 rf-restinstance narrow Δ T−B +0 pp +259692 +3106 +11.3 +51.0 +0.0084

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-requests 1 100% 100% +0 pp +166732 +7.0 +0.0053
rf-restinstance 1 100% 100% +0 pp +259692 +11.3 +0.0084

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-requests-users-01 loaded_rf_requests_skill: 3/3
  • narrow-restinstance-schema-01 loaded_rf_restinstance_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.18
Harness estimate (OpenRouter list prices) 0.03
Credit remaining on the key 2.6768885629999986

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-appium-login-01 rf-appium narrow treatment 3 100% 100% 0 262165 3270 16.0 41.4 0.0087 (est.) no
narrow-appium-login-01 rf-appium narrow baseline (stored) 3 100% 100% 0 171487 2865 13.7 64.1 0.0055 (est.) –
narrow-appium-login-01 rf-appium narrow Δ T−B +0 pp +90678 +405 +2.3 -22.7 +0.0032

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-appium 1 100% 100% +0 pp +90678 +2.3 +0.0032

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-appium-login-01 loaded_rf_appium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.32
Harness estimate (OpenRouter list prices) 0.05
Credit remaining on the key 2.5300221050000005

Gate

gate: FAIL (2 task(s) checked)
- [tokens] narrow-browser-login-01: mean input tokens +89% (166567 -> 314381; budget +75%)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.05

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-browser-login-01 rf-browser narrow treatment 3 100% 100% 0 314381 2380 18.3 74.9 0.0100 (est.) no
narrow-browser-login-01 rf-browser narrow baseline (stored) 3 100% 100% 0 106383 853 9.0 12.5 0.0034 (est.) –
narrow-browser-login-01 rf-browser narrow Δ T−B +0 pp +207998 +1527 +9.3 +62.4 +0.0066
narrow-selenium-login-01 rf-selenium narrow treatment 3 100% 100% 0 152503 2891 12.7 36.1 0.0050 (est.) no
narrow-selenium-login-01 rf-selenium narrow baseline (stored) 3 100% 100% 0 65093 658 7.7 9.1 0.0021 (est.) –
narrow-selenium-login-01 rf-selenium narrow Δ T−B +0 pp +87410 +2233 +5.0 +27.1 +0.0029

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-browser 1 100% 100% +0 pp +207998 +9.3 +0.0066
rf-selenium 1 100% 100% +0 pp +87410 +5.0 +0.0029

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-browser-login-01 loaded_rf_browser_skill: 3/3
  • narrow-selenium-login-01 loaded_rf_selenium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

manykarim and others added 2 commits November 16, 2026 10:59
The libdoc specs moved from eval/fixtures/*/specs/ to eval/graders/specs/,
and their "source" paths still name the developer's local checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
…33101181)

The task prompts and fixtures changed, so every task hash changed.
N=3, both arms, all tiers, turn factor 1.5. The run cost $4.16 of
OpenRouter spend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.17
Harness estimate (OpenRouter list prices) 0.03
Credit remaining on the key 2.6898757029999985

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-platynui-calculator-01 rf-platynui narrow treatment 3 100% 100% 0 315178 2918 17.7 41.5 0.0100 (est.) no
narrow-platynui-calculator-01 rf-platynui narrow baseline (stored) 3 100% 100% 0 343327 4152 19.0 84.6 0.0110 (est.) –
narrow-platynui-calculator-01 rf-platynui narrow Δ T−B +0 pp -28149 -1234 -1.3 -43.1 -0.0010

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-platynui 1 100% 100% +0 pp -28149 -1.3 -0.0010

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-platynui-calculator-01 loaded_rf_platynui_skill: 3/3

Adversarial tier (reported only, never gates)

None.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant