Give long pipeline steps their own execution budget so a timeout reports failure, not cancelled

[konard]

Summary

timeout-minutes was the only deadline in the pipeline, and GitHub reports a
job killed by it as cancelled, not failed. scripts/check-pipeline-status.sh
turns that into an error only on main; on a pull request it is just a warning,
because there a cancellation is normally a superseded run. So a genuine timeout
on a pull request produced nothing red, and on main the error could not name
the deadline that was blown — no step owned one.

This PR gives the long steps their own deadline, so timeout-minutes becomes a
backstop and never the deadline.

Fixes #60

Changes

• scripts/run-with-budget-warning.sh — runs a command under an execution
  budget:
  • set -m puts the command in its own process group, so pytest -n
    workers and pip children are signalled too instead of being orphaned onto
    the runner (which is also why timeout(1) alone is insufficient);
  • ::warning at 70% of the budget, while the overrun can still be acted on;
  • at the deadline: SIGTERM to the group, a grace period, then SIGKILL;
  • ::error title=<label> exceeded its execution budget::… and exit 124,
    matching timeout(1), so the job reports failure naming the budget;
  • BUDGET_ENFORCE=false warns without killing (local runs), and
    CI_VERBOSE=true adds a heartbeat naming what is still running.
• release.yml — lint and test budget installation separately from
  the work that follows, because a slow dependency resolve is the most common
  cause of a Python job overrun and is exactly the part a pytest budget cannot
  see: install 300s, secretlint 300s (cap 20m), install 300s + pytest 900s
  (cap 30m). The two Docker builds run through uses: actions, which a shell
  wrapper cannot own, so they get a step-level timeout-minutes: 40 under their
  60m caps — GitHub reports a step killed by that as a failed step.

Reproduction and verification

The issue's repro is a step allowed to run into the job clock:

    timeout-minutes: 1
    steps:
      - run: sleep 120     # conclusion: cancelled, only a ::warning on a PR

Under the wrapper the same overrun exits 124 with an ::error, so the job
concludes failure. tests/test_run_with_budget_warning.py asserts exactly
that, and that the process tree really dies:

$ bash scripts/run-with-budget-warning.sh 2 "Runaway suite" sleep 120
::warning title=Runaway suite is approaching its timeout::…
::error title=Runaway suite exceeded its execution budget::The command was terminated
after 2s, its full 2s execution budget. …
$ echo $?
124

Tests

• tests/test_run_with_budget_warning.py — exit 124 and the ::error
  annotation on overrun, the whole process group terminated (no zombie
  survivors), the warning arriving while the command is alive, exit-code
  passthrough, the BUDGET_ENFORCE=false escape hatch, the off-by-default
  heartbeat, and rejection of a call with no command.
• tests/test_workflows.py — the invariant that finds the next occurrence:
  every step deadline in every workflow must sum to at most 70% of the
  timeout-minutes it sits under, leaving the remainder for unbudgeted setup;
  plus every job must declare a cap at all, and the budgeted steps must actually
  invoke the wrapper.

All 85 tests pass locally, along with ruff check, ruff format --check and
mypy src.

[konard]

Working session summary

Done. PR #61 is ready and green: #61

Root cause. timeout-minutes was the pipeline's only deadline, and GitHub reports a job it kills as cancelled, not failed. check-pipeline-status.sh:29 only escalates a cancellation to an error on main — on a pull request it stays a ::warning, because there a cancellation is normally a superseded run. So a real timeout on a PR produced nothing red, and no step owned a budget that could name what ran long.

Fix.

• scripts/run-with-budget-warning.sh owns the deadline: set -m so the command gets its own process group (pytest -n workers and pip children die with it — the reason timeout(1) alone isn't enough), ::warning at 70 % of the budget, SIGTERM → grace → SIGKILL at the deadline, ::error title=<label> exceeded its execution budget::…, exit 124. BUDGET_ENFORCE=false warns without killing for local runs; CI_VERBOSE=true adds a heartbeat.
• release.yml: lint budgets install (300 s) and secretlint (300 s) under its 20 m cap; test budgets install (300 s) separately from pytest (900 s) under 30 m — a slow resolve is exactly what a pytest budget can't see. Both Docker builds are uses: steps a shell wrapper can't own, so they get step-level timeout-minutes: 40 under their 60 m caps; GitHub reports a step killed by that as a failed step.

Tests (85 pass, plus ruff check, ruff format --check, mypy src):

• tests/test_run_with_budget_warning.py — exit 124 + the ::error annotation, the whole process tree terminated (I hit a real subtlety here: the orphaned grandchild is reparented to a non-reaping init and lingers as a zombie, so the assertion treats state Z as terminated), the warning arriving while the command is still alive, exit-code passthrough, the escape hatch, the off-by-default heartbeat.
• tests/test_workflows.py — the invariant that catches the next occurrence: every step deadline in every workflow must sum to ≤ 70 % of the job cap above it, and every job must declare a cap at all.

CI confirms the wrapper is live: Run tests took 19s of its 900s execution budget.

---

This summary was automatically extracted from the AI working session output.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $2.502964

📊 Context and tokens usage:

• 80.1K / 1M (8%) input tokens, 25.0K / 128K (20%) output tokens

Total: (86 new + 65.9K cache writes + 2.4M cache reads) input tokens, 25.0K output tokens, $2.502964 cost

🤖 Models used:

• Tool: Anthropic Claude Code
• Requested: opus (claude-opus-5)
• Thinking level: off (disabled)
• Model: Claude Opus 5 (claude-opus-5)

📎 Log file uploaded as Gist (1223KB)

• View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

• All CI checks have passed

---

Auto-merged by hive-mind with --auto-merge flag