Skip to content

verifier: read cicd_check_id when budgeting a T3 case - #89

Open
reacher-z wants to merge 1 commit into
mainfrom
verifier/cicd-check-budget
Open

reacher-z wants to merge 1 commit into
mainfrom
verifier/cicd-check-budget

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Closes #49

case_wall_clock_budget looked up case["check_id"], but manifest cases name their check cicd_check_id, which _cicd already reads. Every T3 case got the 780 s budget of a case with no declared timeout instead of one derived from its check's timeout_sec.

medium check with timeout_sec: 379 Budget
before 780 s
after 6,124 s

Current checks declare 300 to 379 s, so no reward changes today. A check declaring more than about 760 s would have been abandoned inside its own timeout.

Release

The one-line change is applied to all 70 copies of formal.py, which stay byte-identical. It changes the tasks' content hashes, so it ships with the next task release; the release index is untouched here.

Verification

  • Budget computed with the medium manifest and CI/CD suite: 6,124 s for the 379 s check.

  • scripts/check_tasks.py --code-only: 70 PASS.

  • uv run python -m unittest discover -s tests

  • uv run ruff check scripts src tests

  • uv run ruff format --check scripts src tests

case_wall_clock_budget looked up case["check_id"], but manifest cases
name their CI/CD check cicd_check_id (as _cicd reads it). The lookup
never matched, so every T3 case got the 780 s budget of a case with no
declared timeout instead of one derived from its check's timeout_sec
(6,124 s for a 379 s check). Current checks declare 300 to 379 s and fit
under 780 s; a check declaring more than about 760 s would have been
abandoned as an infrastructure failure inside its own timeout.

The same one-line change is applied to all 70 copies of formal.py, which
stay byte-identical. It changes the tasks' content hashes, so it ships
with the next task release.
@reacher-z reacher-z added bug Something isn't working verifier Scoring, capture and check execution in tests/websitebench labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working verifier Scoring, capture and check execution in tests/websitebench

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Case time budget looks up check_id, but T3 cases carry cicd_check_id

1 participant