Repository navigation
Conversation
PlatynUI 0.12.0.dev330 needs Python 3.12+, but the fixture declared requires-python >=3.10. uv then can't resolve the project, so `uv run` in an eval workspace fails and agents can't use the fixture's own environment. Raise the floor to 3.12 and add a test for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Agents read every file in a fixture, and several told them they were in an
eval: smoke tests "irrelevant to any eval task", the sut-failing README
naming the bug and the shortcuts its graders reject, sut-legacy-style
explaining the adversarial trap, sut-selenium naming the "adversarial
element count task". The PlatynUI and AppiumLibrary prompts sent agents to
a committed libdoc spec ("a libdoc spec is in specs/"), so they parsed JSON
instead of checking the installed library as a user would.
- Move the specs to eval/graders/specs (grader data, never staged);
keywords_resolve resolves a relative spec_dir against the repository.
- The five spec tasks ask agents to check keywords against the installed
library.
- Fixture READMEs, smoke tests and pyproject descriptions describe the
project only; maintainer notes stay in eval/fixtures/README.md.
- narrow-non-rf-control-01 no longer tells the agent to avoid RF tooling
(that made the control meaningless); narrow-rf-mcp-execute-01 no longer
calls itself an evaluation.
- Tests: fixtures contain no eval wording and no specs; spec_dir works.
Task and fixture hashes change, so the GLM baselines need a re-baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 36 · pass: 36 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.26 · turn-limit stops: 4 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.06 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.05 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
The libdoc specs moved from eval/fixtures/*/specs/ to eval/graders/specs/, and their "source" paths still name the developer's local checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
…33101181) The task prompts and fixtures changed, so every task hash changed. N=3, both arms, all tiers, turn factor 1.5. The run cost $4.16 of OpenRouter spend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)Model: OpenRouter spend
Gaterf-skill-eval reportRuns: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03 Pass rate = runs whose gate result is Tasks
Skills
Skipped / errored checksNone. Incomplete runsNone. Process checks (treatment arm, not in deltas)
Adversarial tier (reported only, never gates)None. |
Summary
Agents read every file of a fixture, and several of them told the agent it was being evaluated, or gave away the trap of an adversarial task. The PlatynUI and AppiumLibrary prompts sent agents to a committed libdoc spec, so they parsed JSON instead of checking the installed library. That is what led to the misleading robotcodedev/robotcode#689.
Stacked on #19 (its commit is the first one here). Merge #19 first.
What was unrealistic
sut-failing/README.mdadv-make-it-pass).sut-legacy-style/README.mdsut-selenium/README.mdsut-pylib,sut-language-*,sut-api,sut-minimal,sut-browserREADMEspyproject.tomldescriptionsnarrow-non-rf-control-01narrow-rf-mcp-execute-01Changes
eval/fixtures/sut-{appium,platynui}/specs/→eval/graders/specs/(grader data, never staged, likeeval/graders/language/).keywords_resolveresolves a relativespec_diragainst the repository; the tasks usespec_dir: eval/graders/specs. The prompts ask agents to check keywords against the installed library. The versions matchuv.lock(AppiumLibrary 3.2.1, PlatynUI 0.12.0.dev330).eval/fixtures/README.md, which isn't staged.test_fixture_reads_like_a_real_projectchecks every fixture file for eval wording;test_fixtures_ship_no_libdoc_specs;spec_dirtest.eval/graders/specs/README.md, thekeywords_resolverow in the tasks README, the fixture checklist, anddocs/ci/desktop-live-check.md.Baselines
Every task and fixture hash changes, so the gate reports
rebaseline-neededuntil the GLM 5.3 Flash baselines are re-recorded. A re-baseline run is dispatched on this branch, and its baselines will be committed here.Testing
pytest tests/eval: 665 passed. The 4 Browser tests that fail locally needrfbrowser init; CI runs it.ruff,mypyandvalidate-tasks(all plugins): clean.check-docs-voice: OK.🤖 Generated with Claude Code
https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP