Skip to content

feat(verify): configurable per-command verify timeout ([verify] command_timeout_min) - #897

Merged
pbean merged 1 commit into
mainfrom
feat/857-verify-command-timeout
Oct 13, 2026
Merged

pbean merged 1 commit into
mainfrom
feat/857-verify-command-timeout

Conversation

@pbean

@pbean pbean commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Every [verify] commands entry was killed after a fixed 30 minutes (verify.COMMAND_TIMEOUT_S). This adds [verify] command_timeout_min (default 30, must be >= 1) so a slow but healthy gate, such as a mutation suite, can be given a larger bound without patching the installed package.

  • policy.py: new VerifyPolicy.command_timeout_min, parsed with _typed_int, rejected below 1, documented in the emitted template.
  • verify.py: COMMAND_TIMEOUT_S is gone. run_verify_commands passes command_timeout_min * 60 to each command. _probe_timed_out is renamed _timed_out_leg, since [verify] classification now uses it as well.
  • core.toml schema entry, regenerated docs/reference/configuration.md, the FEATURES config line, and the runsetup digest docstring. The field is not added to the config digest because it bounds a command and cannot name a program.

Design notes

  • Global only. There is no per-command override. commands is a list of strings, and the TUI lines editor would flatten table entries on save.
  • Neutral wording for the agent. A timed-out command's failure reason now ends with timed out after N min (the per-command verify time limit) instead of a bare timed out. The first line, verify command failed (rc=-1): <cmd>, is byte-identical and the failure stays fixable. This text is written verbatim into the repair session's feedback, so it never names policy.toml and never suggests raising the limit. An agent edit there would fall outside dirty-tree accounting and outside the config digest.
  • --reverify names the key. This output goes to the operator, so it reads '<cmd>' timed out after N min — raise [verify] command_timeout_min in policy.toml if this gate is legitimately this slow.
  • The classifier keyword defaults to None, so existing direct calls keep today's exact text.

Test plan

  • New tests:
    • policy: default/parse/template, >= 1 (0 and -5), non-integers (true, 2.5, "60")
    • verify: the bound reaches the runner ([2700, 2700] and the default [1800])
    • verify: the neutral timeout reason, plus SIGHUP, spawn-error and no-keyword negative rows
    • the review composition passes the limit
    • --reverify names the limit and the key (420 s)
    • the dev-retry feedback names the limit and not policy.toml
  • Each new test ablated: its gate was removed, the test went red, and the gate was restored (9 ablations).
  • Three existing tests that monkeypatched the deleted constant were rewired to wrap _run_shell_command.
  • uv run pytest -q -n logical: 14427 passed, 113 skipped.
  • uv run pyright: 0 errors.
  • trunk check: no issues.

Closes #857

Summary by CodeRabbit

  • New Features
    • Verification commands now have a configurable per-command timeout, set to 30 minutes by default.
  • Bug Fixes
    • Timeout failures now identify the command and configured limit. Retry guidance points to the setting to adjust, and timed-out commands count as ordinary verification failures.

@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 606a85de-3bed-42b7-87ac-8bbab8b2538a
📥 Commits

Reviewing files that changed from the base of the PR and between 7834dfa and b4e2d24.

📒 Files selected for processing (13)
  • CHANGELOG.md
  • docs/FEATURES.md
  • docs/reference/configuration.md
  • src/bmad_loop/cli.py
  • src/bmad_loop/data/settings/core.toml
  • src/bmad_loop/engine.py
  • src/bmad_loop/policy.py
  • src/bmad_loop/runsetup.py
  • src/bmad_loop/verify.py
  • tests/test_cli.py
  • tests/test_engine.py
  • tests/test_policy.py
  • tests/test_verify.py

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

Walkthrough

Verification commands now use a configurable per-command timeout, defaulting to 30 minutes. Timed-out commands remain fixable verification failures, and failure messages report the configured limit.

Changes

Verify timeout configuration

Layer / File(s) Summary
Policy, validation, and configuration references
src/bmad_loop/policy.py, src/bmad_loop/data/settings/core.toml, src/bmad_loop/runsetup.py, docs/reference/configuration.md, docs/FEATURES.md, CHANGELOG.md
The verify policy adds command_timeout_min, defaulting to 30, and rejects values below 1 or values that are not integers. The setting is added to the configuration schema and digest. Documentation describes its timeout behavior.
Command execution and timeout classification
src/bmad_loop/verify.py, src/bmad_loop/engine.py, tests/test_verify.py, tests/test_engine.py
Verification commands use the configured timeout in seconds. Timeout results report the configured limit and remain fixable failures. Environment-probe timeout checks use the shared timeout predicate. Tests cover configuration, classification, and repair feedback.
Reverify timeout reporting
src/bmad_loop/cli.py, tests/test_cli.py, CHANGELOG.md
The reverify failure message names the timed-out command, the configured limit, and the policy setting to adjust. A CLI test checks the reported limit and setting name.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant run_verify_commands
  participant _run_shell_command
  participant Engine
  participant verify_command_results_outcome
  run_verify_commands->>_run_shell_command: Run command with configured timeout in seconds
  _run_shell_command-->>run_verify_commands: Return command result
  Engine->>verify_command_results_outcome: Classify results with configured timeout in minutes
Loading

Suggested reviewers: dracic

Merge Risk: ⚪ Minimal · up to b4e2d

The configurable timeout can be changed without the digest-based interruption in question. No actionable merge-blocking issue remains after normal checks.

Pre-merge checks · Passed 5
Passed Passed checks (5 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely identifies the main change: adding a configurable per-command verification timeout through [verify] command_timeout_min.
Linked Issues check Passed Issue #857 requires a configurable timeout for [verify] commands. The PR adds VerifyPolicy.command_timeout_min with default 30 and validation of values >= 1. run_verify_commands uses the configured va…
Out of Scope Changes check Passed The changes remain connected to issue #857. Policy parsing, schema updates, documentation, timeout classification, operator timeout reporting, digest documentation, and tests support the configurable …
Docstring Coverage Passed Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 4 files. (9 skipped: 4 u…
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit reads the timeout line,
Thirty minutes set by default,
A longer gate can have more time,
A timed-out run reports its limit,
The policy shows the key to raise,
Then hops away through logs of hay.

Comment @coderabbitai help to get the list of available commands.

@pbean

pbean commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T15:02:54.523148Z b4e2d24 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: b4e2d2472e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pbean

pbean commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@pbean
pbean merged commit 5cec451 into main Oct 13, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify command timeout is hard-coded at 30 minutes; make it configurable

1 participant