Repository navigation
feat: find flaky checks (prw flaky, MCP get_flaky_checks) - #22
Merged
Merged
Conversation
A job that failed and then passed when its workflow run was re-run is flaky:
same commit, same code, different result. `prw flaky <owner/repo | URL>`
scans the repository's last --runs workflow runs (default 300, max 1000),
fetches the jobs of every attempt of the re-run ones, and reports each job
that failed (failure / timed_out) in one attempt and succeeded in a later one,
grouped by workflow and job: how many runs it flaked in, when last, and the
latest failed attempt's link. Cancelled jobs (fail-fast victims) don't count;
jobs that kept failing aren't flaky. `--json` for scripts.
The MCP tool `get_flaky_checks` gives agents the same, so they can tell a
known-flaky failure from one their change caused.
GitHub takes seconds per runs/jobs request on large repositories, so pages
and attempts are fetched in parallel (4 at a time) and runs are listed
without their pull_requests payload: grafana/grafana went from 24s to 9s.
Verified live: grafana/grafana (4 flaky jobs in 300 runs), hashicorp/terraform
("Unit Tests"), devblac/prw (none) and over MCP stdio.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap #16. A job that failed and then passed when its workflow run was re-run — same commit, same code — is flaky.
prw flakyfinds them.How
--runsworkflow runs (default 300, max 1000); for each run withrun_attempt > 1, fetches the jobs of every attempt.failure/timed_out) in one attempt and succeeded in a later one. Cancelled jobs (fail-fast victims) don't count; jobs that kept failing aren't flaky. Counted once per run.--json.owner/repoor any GitHub URL in the repo (e.g. a PR URL).get_flaky_checks: lets an agent tell a known-flaky failure from one its change caused.exclude_pull_requests=true. grafana/grafana: 24 s → 9 s.prw watch --mine) #16 was dropped: low value for its cost.Test plan
go test ./... -race,go vet ./...Quick Checks / Unit Tests), devblac/prw (none), MCP over stdioStacked on #21.
🤖 Generated with Claude Code