Skip to content

Add websitebench summarize for Harbor job directories - #70

Open
reacher-z wants to merge 1 commit into
mainfrom
feat/summarize-jobs
Open

reacher-z wants to merge 1 commit into
mainfrom
feat/summarize-jobs

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Closes #9

Harbor 0.25 removed harbor job summarize. websitebench summarize JOB_DIR ... reads verifier/reward.txt and verifier/eval.json in every trial directory and prints:

$ websitebench summarize jobs/<job>
task    status        passed  reward
medium  VALID_RUN        412  0.6338
notion  INVALID_RUN           -  (CANDIDATE_ABSENT)

mean reward over 1 scored trial: 0.6338
1 trial has no reward and is left out

passed is the case_points from eval.json; the reason in parentheses comes from the same file. A trial directory without verifier output is listed as NOT_VERIFIED.

Changes

  • New src/websitebench/summary.py (Trial, read_trial, summarize).
  • cli.py gains the summarize subcommand; docs/running.md shows it.

Tests

tests/test_summary.py covers scored, invalid and unverified trials, singular and plural counts, an empty job and a missing directory.

  • uv run python -m unittest discover -s tests
  • uv run ruff check scripts src tests
  • uv run ruff format --check scripts src tests

Harbor 0.25 removed harbor job summarize, so reading a run meant opening
reward.txt and eval.json in every trial directory. summarize prints one
row per trial with its status, cases passed and reward, the mean reward
over the scored trials, and the reason recorded for each trial that has
no reward.
@reacher-z reacher-z added enhancement New feature or request cli The websitebench command and download path labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli The websitebench command and download path enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Summarize rewards across a Harbor job directory

1 participant