Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 74 additions & 74 deletions docs/tasks.md
Original file line number Diff line number Diff line change
@@ -1,80 +1,80 @@
# Tasks

WebsiteBench has 70 tasks with 46,916 hidden test cases in total. Each row lists the task's case count and the size of the complete task download. Every task requests 8 CPUs, 8 GiB of memory and 20 GiB of disk for each agent and verifier environment, with a 20-minute build timeout. Every task gives the agent 16 hours and the verifier 8 hours.
WebsiteBench has 70 tasks with 46,916 hidden test cases in total. Each row lists the website the task reproduces, the task's case count and the size of the complete task download. Every task requests 8 CPUs, 8 GiB of memory and 20 GiB of disk for each agent and verifier environment, with a 20-minute build timeout. Every task gives the agent 16 hours and the verifier 8 hours.

Start with `medium`: it is small to download.
Start with `medium`, a reproduction of Medium: it is small to download.

| Task | Cases | Download |
| --- | ---: | ---: |
| `1password-web` | 679 | 8 MB |
| `33` | 765 | 73 MB |
| `ace-hardware` | 433 | 17 MB |
| `airtable` | 589 | 22 MB |
| `amazon` | 1,025 | 218 MB |
| `amazon-home-services` | 799 | 17 MB |
| `amc-theatres` | 369 | 17 MB |
| `angi` | 1,000 | 46 MB |
| `apple` | 700 | 421 MB |
| `article` | 1,759 | 254 MB |
| `asana` | 346 | 354 MB |
| `aspca-pet-insurance` | 1,019 | 145 MB |
| `autoslash` | 353 | 43 MB |
| `autotrader` | 487 | 9 MB |
| `backcountry` | 1,317 | 590 MB |
| `bean-box` | 987 | 5 MB |
| `beeradvocate` | 446 | 6 MB |
| `blinkist` | 437 | 4 MB |
| `blueapron` | 410 | 85 MB |
| `bluemercury` | 694 | 95 MB |
| `boardgamegeek` | 1,020 | 47 MB |
| `bookshop` | 632 | 63 MB |
| `breakoutgames-2` | 387 | 27 MB |
| `brooklinen` | 736 | 93 MB |
| `capterra` | 632 | 162 MB |
| `change` | 420 | 45 MB |
| `columbia` | 338 | 360 MB |
| `crumbl-cookies` | 531 | 179 MB |
| `crunchyroll` | 681 | 22 MB |
| `curology` | 539 | 34 MB |
| `dooney-bourke` | 754 | 10 MB |
| `eatthismuch` | 668 | 2.4 GB |
| `etsy` | 347 | 19 MB |
| `eventbrite` | 1,655 | 44 MB |
| `everydollar` | 394 | 11 MB |
| `everyplate` | 2,016 | 72 MB |
| `fandango` | 387 | 430 MB |
| `freshdirect` | 990 | 7 MB |
| `hipcamp` | 302 | 68 MB |
| `hmart` | 714 | 161 MB |
| `imdb` | 509 | 463 MB |
| `insureon` | 522 | 41 MB |
| `insurify` | 350 | 53 MB |
| `ioby` | 380 | 85 MB |
| `kanetix-ca` | 809 | 40 MB |
| `leetcode` | 326 | 14 MB |
| `legalnature` | 389 | 336 MB |
| `linkedin` | 565 | 42 MB |
| `llbean` | 423 | 149 MB |
| `lowes` | 494 | 26 MB |
| `medium` | 650 | 7 MB |
| `meetup` | 347 | 111 MB |
| `notion` | 393 | 12 MB |
| `ollie` | 394 | 56 MB |
| `ontario-parks` | 313 | 4 MB |
| `opentable` | 512 | 21 MB |
| `petco` | 609 | 24 MB |
| `petfinder` | 1,783 | 450 MB |
| `petsmart` | 837 | 42 MB |
| `ratemyprofessors` | 721 | 7 MB |
| `resy` | 519 | 111 MB |
| `simplify-jobs` | 752 | 5 MB |
| `target` | 537 | 193 MB |
| `taskrabbit` | 871 | 123 MB |
| `tripadvisor` | 625 | 17 MB |
| `tripit` | 478 | 49 MB |
| `ubereats` | 338 | 25 MB |
| `warbyparker` | 992 | 288 MB |
| `wayfair` | 1,163 | 7 MB |
| `zennioptical` | 558 | 146 MB |
| Task | Site | Cases | Download |
| --- | --- | ---: | ---: |
| `1password-web` | 1Password Web | 679 | 8 MB |
| `33` | Coursera | 765 | 73 MB |
| `ace-hardware` | Ace Hardware | 433 | 17 MB |
| `airtable` | Airtable | 589 | 22 MB |
| `amazon` | Amazon | 1,025 | 218 MB |
| `amazon-home-services` | Amazon Home Services | 799 | 17 MB |
| `amc-theatres` | AMC Theatres | 369 | 17 MB |
| `angi` | Angi | 1,000 | 46 MB |
| `apple` | Apple | 700 | 421 MB |
| `article` | Article | 1,759 | 254 MB |
| `asana` | Asana | 346 | 354 MB |
| `aspca-pet-insurance` | ASPCA Pet Health Insurance | 1,019 | 145 MB |
| `autoslash` | AutoSlash | 353 | 43 MB |
| `autotrader` | Autotrader | 487 | 9 MB |
| `backcountry` | Backcountry | 1,317 | 590 MB |
| `bean-box` | Bean Box | 987 | 5 MB |
| `beeradvocate` | BeerAdvocate | 446 | 6 MB |
| `blinkist` | Blinkist | 437 | 4 MB |
| `blueapron` | Blue Apron | 410 | 85 MB |
| `bluemercury` | bluemercury | 694 | 95 MB |
| `boardgamegeek` | BoardGameGeek | 1,020 | 47 MB |
| `bookshop` | Bookshop.org | 632 | 63 MB |
| `breakoutgames-2` | Breakout Games | 387 | 27 MB |
| `brooklinen` | Brooklinen | 736 | 93 MB |
| `capterra` | Capterra | 632 | 162 MB |
| `change` | Change.org | 420 | 45 MB |
| `columbia` | Columbia Sportswear | 338 | 360 MB |
| `crumbl-cookies` | Crumbl Cookies | 531 | 179 MB |
| `crunchyroll` | Crunchyroll | 681 | 22 MB |
| `curology` | Curology | 539 | 34 MB |
| `dooney-bourke` | Dooney & Bourke | 754 | 10 MB |
| `eatthismuch` | Eat This Much | 668 | 2.4 GB |
| `etsy` | Etsy | 347 | 19 MB |
| `eventbrite` | Eventbrite | 1,655 | 44 MB |
| `everydollar` | EveryDollar | 394 | 11 MB |
| `everyplate` | EveryPlate | 2,016 | 72 MB |
| `fandango` | Fandango | 387 | 430 MB |
| `freshdirect` | FreshDirect | 990 | 7 MB |
| `hipcamp` | Hipcamp | 302 | 68 MB |
| `hmart` | H Mart | 714 | 161 MB |
| `imdb` | IMDb | 509 | 463 MB |
| `insureon` | Insureon | 522 | 41 MB |
| `insurify` | Insurify | 350 | 53 MB |
| `ioby` | ioby | 380 | 85 MB |
| `kanetix-ca` | Kanetix / Rates.ca | 809 | 40 MB |
| `leetcode` | LeetCode | 326 | 14 MB |
| `legalnature` | LegalNature | 389 | 336 MB |
| `linkedin` | LinkedIn | 565 | 42 MB |
| `llbean` | L.L.Bean | 423 | 149 MB |
| `lowes` | Lowe's | 494 | 26 MB |
| `medium` | Medium | 650 | 7 MB |
| `meetup` | Meetup | 347 | 111 MB |
| `notion` | Notion | 393 | 12 MB |
| `ollie` | Ollie | 394 | 56 MB |
| `ontario-parks` | Ontario Parks Reservations | 313 | 4 MB |
| `opentable` | OpenTable | 512 | 21 MB |
| `petco` | Petco | 609 | 24 MB |
| `petfinder` | Petfinder | 1,783 | 450 MB |
| `petsmart` | PetSmart | 837 | 42 MB |
| `ratemyprofessors` | Rate My Professors | 721 | 7 MB |
| `resy` | Resy | 519 | 111 MB |
| `simplify-jobs` | Simplify Jobs | 752 | 5 MB |
| `target` | Target | 537 | 193 MB |
| `taskrabbit` | TaskRabbit | 871 | 123 MB |
| `tripadvisor` | Tripadvisor | 625 | 17 MB |
| `tripit` | TripIt | 478 | 49 MB |
| `ubereats` | Uber Eats | 338 | 25 MB |
| `warbyparker` | Warby Parker | 992 | 288 MB |
| `wayfair` | Wayfair | 1,163 | 7 MB |
| `zennioptical` | Zenni Optical | 558 | 146 MB |

Total download: 9.7 GB.
Loading