Skip to content

Add ClawBench to GUI Agent benchmarks #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
reacher-z wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add ClawBench to GUI Agent benchmarks #8

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Save hours of research time—get straight to what matters.
| Benchmark | Link | Year | Highlight | Data Size | Metric | Measurement | Platform |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MiniWoB++ | https://github.com/Farama-Foundation/miniwob-plusplus | 2017 | Evaluates agents on basic web interactions like clicking, typing, and form navigation. | 100 web interaction tasks | Task Success Rate | Element Match | Web |
| ClawBench | https://github.com/TIGER-AI-Lab/ClawBench | 2026 | Tests whether agents can complete everyday online tasks (purchases, bookings, job applications, email) on live production websites, using a Chrome-extension + CDP layer that intercepts only the final write request for safe end-to-end runs. | 153 tasks across 144 live websites, 15 categories | Task Success Rate | Execution-based (blocked final submission) | Web |
| RUSS | https://github.com/xnancy/russ | 2021 | Uses ThingTalk for mapping natural language to web actions, enabling precise web-based task execution in real HTML environments. | 741 instructions | Task Success Rate | Text Match, Element Match | Web |
| WebShop | https://webshop-pnlp.github.io | 2022 | Simulates e-commerce navigation with real-world products, challenging agents with instruction comprehension, multi-page navigation, and strategic exploration. | 12,087 instructions | Task Success Rate, Step Success Rate* | Text Match | Web |
| Mind2Web | https://github.com/OSU-NLP-Group/Mind2Web | 2023 | Tests adaptability on real-world, dynamic websites across domains. | 2,000 tasks | Step Success Rate, Task Success Rate | Element Match, Action Match | Web |
Expand Down