Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Contributing

## Where changes go

| Change | Where |
| --- | --- |
| The `websitebench` command and release index | `src/websitebench/` |
| Source task checks | `scripts/check_tasks.py` |
| Tests for both | `tests/` |
| Documentation | `README.md`, `docs/` |
| Task definitions, verifier and fixtures | `tasks/<task>/` |

`src/websitebench/websitebench-v1.json` pins the Harbor content hash of every file in each v1 task. Harbor Hub and Hugging Face serve exactly those files, so an edit under `tasks/` reaches users only with a new task release and index. Open an issue for a task or verifier problem first; the task problem form asks for the details needed to reproduce it.

## Development setup

```sh
uv sync --locked --group dev
uv run python -m unittest discover -s tests
uv run ruff check scripts src tests
uv run ruff format --check scripts src tests
uv run python scripts/check_tasks.py --code-only
```

[Development](docs/development.md) describes each check and the release process.

## Pull requests

- Keep each pull request to one change, with a test for behavior changes.
- Write commit subjects in the imperative mood, for example "Reject an --output that is a file".
- Tests run offline: mock `harbor` and `hf` rather than downloading tasks.
- Update the docs in the same pull request as a user-visible change.

## Reporting

Use the issue forms for bugs, task problems and feature requests. Report a way to raise a reward without rebuilding the site, read hidden tests, or escape the agent or candidate sandbox privately, as described in [SECURITY.md](SECURITY.md).
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@
| [Tasks](tasks.md) | Every task's cases and download size |
| [Task format](task-format.md) | Task directories, compile and runtime interface, scoring |
| [Development](development.md) | Checks, tests, CI, package releases |
| [Contributing](../CONTRIBUTING.md) | Where changes go, pull requests, reporting |
Loading