Skip to content

Add websitebench verify to check local task copies - #69

Open
reacher-z wants to merge 1 commit into
mainfrom
feat/verify-command
Open

reacher-z wants to merge 1 commit into
mainfrom
feat/verify-command

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Closes #8

websitebench verify [DIR ...] recomputes the Harbor content hash of each task directory, checks the executable bits the release index lists, and prints OK <task> or FAIL <reason> per task. A task edited after download reports:

$ websitebench verify local-tasks/notion
FAIL notion: content hash ea6a53388d46 over 84 files does not match the release (8f279b5cbc47 over 397 files)
1 of 1 tasks differ from the release

With no arguments it checks the release tasks found in ./local-tasks. Each directory must be named after its task. Exit status is 1 when any task differs, and 2 (usage error) when there is nothing to check.

Changes

  • download.check_executables() reports listed files that are missing or not executable.
  • cli.py gains the verify subcommand.
  • docs/pypi.md and docs/running.md describe it.

Tests

New tests: a matching task, the default directory, a changed file, a listed executable without its mode, a directory that is not a release task, and nothing to verify.

  • uv run python -m unittest discover -s tests
  • uv run ruff check scripts src tests
  • uv run ruff format --check scripts src tests

A local task copy can drift from the release after an interrupted copy,
an rsync without -p, or an edit made while debugging. verify recomputes
the Harbor content hash of each task directory, checks the executable
bits the release lists, and exits 1 when any task differs. With no
arguments it checks the release tasks found in ./local-tasks.
@reacher-z reacher-z added enhancement New feature or request cli The websitebench command and download path labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli The websitebench command and download path enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a command that checks a local task copy against the release

1 participant