Skip to content

check_tasks: fail when a shared verifier module drifts between tasks - #76

Open
reacher-z wants to merge 1 commit into
mainfrom
check-tasks/shared-verifier-drift
Open

reacher-z wants to merge 1 commit into
mainfrom
check-tasks/shared-verifier-drift

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Every task ships its own copy of the verifier package. The 26 modules under tests/websitebench/ that all 70 tasks carry are byte-identical, which keeps rewards comparable across tasks. A fix applied to some copies and not the others would go unnoticed.

Changes

  • scripts/check_tasks.py gains shared_verifier_drift(tasks): it hashes each file under tests/websitebench/ that every checked task has and reports a differing copy as
    FAIL shared verifier: <path> differs from the copy in N tasks in: <tasks>.
  • It runs whenever more than one task is checked. Task-specific modules (for example t3_checks.py in three tasks) are left alone.
  • docs/development.md mentions the check.

Tests

New SharedVerifierDrift tests. Editing one copy of a shared module in notion makes the check fail and name notion; the unmodified tree passes.

  • uv run python -m unittest discover -s tests
  • uv run ruff check scripts src tests
  • uv run ruff format --check scripts src tests
  • uv run python scripts/check_tasks.py --code-only

Every task ships its own copy of the verifier package. The 26 modules
that all 70 tasks carry are byte-identical today, which is what makes a
reward on one task comparable with a reward on another. A fix applied to
some copies and not the others would break that silently. When more than
one task is checked, report each shared module whose copies differ and
name the tasks that hold the minority copy. Task-specific modules, such
as t3_checks.py in three tasks, are left alone.
@reacher-z reacher-z added tasks Task definitions and payloads verifier Scoring, capture and check execution in tests/websitebench labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tasks Task definitions and payloads verifier Scoring, capture and check execution in tests/websitebench

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant