Skip to content

Test that the release index lists exactly the executable task files - #78

Open
reacher-z wants to merge 1 commit into
mainfrom
test/index-executables
Open

reacher-z wants to merge 1 commit into
mainfrom
test/index-executables

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

restore_modes sets the executable bit only on the files the release index lists, because Hugging Face does not keep file modes. Nothing checked that list against the checkout, so a script made executable in tasks/ but missing from the index would install from Hugging Face without its executable bit.

The new test walks every file in tasks/<task> and checks that it is executable exactly when the task's executables in websitebench-v1.json lists it. All 70 tasks agree. Clearing the bit on tasks/medium/environment/reference/run.sh, or setting it on tasks/medium/instruction.md, makes the test fail.

  • uv run python -m unittest discover -s tests
  • uv run ruff check scripts src tests
  • uv run ruff format --check scripts src tests

restore_modes makes executable only the files the index lists, because
Hugging Face does not keep file modes. Nothing checked that the list
agrees with the checkout, so a script made executable in tasks/ but left
out of the index would install from Hugging Face without its executable
bit. The index and the source modes agree for all 70 tasks today; the
test keeps it that way.
@reacher-z reacher-z added tasks Task definitions and payloads packaging PyPI package and release index labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

packaging PyPI package and release index tasks Task definitions and payloads

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant