diff --git a/CHANGELOG.md b/CHANGELOG.md index d6030e3..2842d97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `prw run` now notifies once (`merged` or `closed`) and automatically stops watching a PR once it is merged or closed, instead of polling it forever. This notification always fires, regardless of `--on`/`notification_filter`. ### Added +- `prw flaky `: finds flaky GitHub Actions jobs — jobs that failed and then passed when their workflow run was re-run on the same commit — in the repository's last `--runs` workflow runs (default 300, max 1000), grouped by workflow and job with counts, last occurrence and a link to the latest failed attempt; `--json`. Cancelled jobs (fail-fast victims) don't count. New MCP tool `get_flaky_checks`. - Failed job logs: `prw checks --logs` and `prw wait --logs` print, for each failed GitHub Actions job, the output of the step that failed (its last `--log-lines` lines up to the error, timestamps/colors/runner markup stripped); `failed_logs` / `logs` in `--json`. New MCP tool `get_failed_logs` returns the same to agents, and `get_ci_status` / `wait_for_ci` point to it on failure. Checks from other CI systems are listed with their link, as they have no log in the API. - `prw checks `: every check run and commit status of a PR's head commit with its result, duration and link, failures first; `--json` for scripts. Superseded runs are left out. Exits `0` passed, `1` failed, `3` error, `4` no CI, `8` still running. - Review and merge events: `prw run` notifies when a PR is approved, gets changes requested, runs into merge conflicts, or becomes ready to merge after branch protection was holding it back. Shown in console, native, Slack/Discord and generic webhooks (`type` `pr_review_change` / `pr_merge_state_change`), filtered by `--on` (`fail`: changes requested and conflicts; `success`: approved and ready). `prw list` gains `REVIEW` and `MERGE` columns (`review` / `merge` in `--json` and MCP `list_watched`); `prw wait --json` and the MCP tools report `merge_state`. Disable with `prw config set pr_events false`. diff --git a/README.md b/README.md index 7d6f139..86eb893 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,8 @@ The tool is deliberately minimal. It does one thing well: watch PRs and tell you ## Features -- **`prw checks`**: every check of a PR with its result, duration and link +- **`prw checks`**: every check of a PR with its result, duration and link — and with `--logs`, the output of the step that failed +- **`prw flaky`**: find a repository's flaky jobs (failed, then passed on re-run of the same commit) - **`prw wait`**: block until a PR's CI finishes and exit 0/1/2 — for scripts, pipelines and AI coding agents - **MCP server** (`prw mcp`): lets Claude Code, Claude Desktop, Cursor and other agents check and wait for PR CI as tools - **Auto-watch**: `prw watch --mine` / `--review-requested` follow your PRs and review requests across all repositories, no URLs needed @@ -301,6 +302,22 @@ $ prw checks https://github.com/owner/repo/pull/123 --logs `--log-lines` sets how many lines (default 40). At most 5 job logs are downloaded per command. Checks from other CI systems (and legacy commit statuses) have no log in the API, so you get their link instead. Logs expire after GitHub's retention period (90 days by default). Fine-grained tokens need the **Actions: read** permission. +## Finding flaky checks: `prw flaky` + +A job that fails and then passes when its workflow run is re-run — same commit, same code — is flaky. `prw flaky` scans a repository's recent GitHub Actions runs (300 by default, `--runs` up to 1000) for exactly that: + +``` +$ prw flaky grafana/grafana +grafana/grafana: scanned 300 workflow run(s) since 2026-10-24, 3 of them re-run. + +FLAKED WORKFLOW JOB LAST SEEN LATEST FAILED ATTEMPT +1x Backend Unit Tests Grafana Enterprise (3/8) 2026-10-24 15:32 https://github.com/grafana/grafana/actions/runs/…/job/… +1x Integration Tests Sqlite Enterprise (2/16) 2026-10-24 15:25 https://github.com/grafana/grafana/actions/runs/…/job/… +1x End-to-end tests Playwright E2E tests (5/8) 2026-10-24 07:55 https://github.com/grafana/grafana/actions/runs/…/job/… +``` + +It takes a repository (`owner/repo`) or any GitHub URL in it, such as a PR's. Only re-run runs can tell flakiness apart from real failures, and jobs cancelled because another job failed first don't count. `--json` for scripts. It costs a few requests per 100 runs plus a couple per re-run run; on large repositories GitHub takes a few seconds per request, so prw fetches in parallel. + ## Using prw from AI agents: `prw mcp` `prw mcp` runs a [Model Context Protocol](https://modelcontextprotocol.io) server over stdio, so coding agents can check a PR's CI and wait for it after pushing — and read which checks failed — without screen-scraping `gh` output. @@ -327,6 +344,7 @@ Tools: |------|--------------| | `get_ci_status` | Current state of a PR (`success`, `failure`, `error`, `pending`, `merged`, `closed`, `no_ci`), check counts, failed checks with links. Returns immediately. | | `get_failed_logs` | For each failed GitHub Actions job, the output of the step that failed (last `max_lines` lines up to the error, default 40). What an agent needs to fix the failure without a browser. | +| `get_flaky_checks` | A repository's jobs that failed and then passed on re-run of the same commit, with counts and links — so an agent can tell a known-flaky failure from one its change caused. | | `wait_for_ci` | Blocks until CI finishes (`timeout_seconds` default 600, max 3600). Fails fast on the first failed check, follows new pushes, sends progress notifications. On timeout it returns `pending` with `timed_out: true` so the agent can call it again. | | `list_watched` | PRs on the local watch list with their last known state. | | `watch_pr` / `unwatch_pr` | Add or remove a PR from the watch list (a running `prw run` picks it up). | diff --git a/TASKS.md b/TASKS.md index a7bd82b..10db5e1 100644 --- a/TASKS.md +++ b/TASKS.md @@ -95,7 +95,8 @@ Legend — Impact: 1–5 · Effort: S (≤½ day), M (1–2 days), L (3+ days) - ✅ Codex follow-ups on #18/#19 — failed head re-check no longer accepted as "unchanged"; review-request PRs only dropped when proven withdrawn (open or 404), never on a cancelled or failed lookup; all review pages read. - ✅ #12 `prw checks ` — done (buckets fail/cancel/pending/pass/neutral/skipping, durations, links, `--json`, exit 8 while running; verified live on devblac/docker-lint#5, zephyrproject-rtos/zephyr#105183 — 35 checks, superseded run hidden — and devblac/prw#13) - ✅ #15 Failed job log excerpt — done (`--logs` on `checks`/`wait`, MCP `get_failed_logs`; parser tested on a real staticcheck failure log; verified live on cloudposse/atmos#3476, #3422 and #2199 — a commit status — and over MCP stdio) -- Next: #16 flakiness detection + status history +- ✅ #16 Flakiness detection — done as `prw flaky` + MCP `get_flaky_checks` (failed-then-passed across attempts of the same run; verified live: grafana/grafana 4 flaky jobs in 300 runs, hashicorp/terraform "Unit Tests"; parallel fetching cut grafana from 24s to 9s). The "status history" half was dropped: low value for its cost. +- Next: #17 packaging (Homebrew/Scoop) + keychain token storage, #8 `gh` extension publishing — both need owner decisions ## Planned diff --git a/cmd/prw/flaky.go b/cmd/prw/flaky.go new file mode 100644 index 0000000..213084c --- /dev/null +++ b/cmd/prw/flaky.go @@ -0,0 +1,123 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "io" + "os" + "regexp" + "text/tabwriter" + + "github.com/spf13/cobra" + + "github.com/devblac/prw/internal/config" + "github.com/devblac/prw/internal/github" +) + +var ( + flakyRuns int + flakyJSON bool +) + +func init() { + rootCmd.AddCommand(flakyCmd) + flakyCmd.Flags().IntVar(&flakyRuns, "runs", github.DefaultFlakyRuns, fmt.Sprintf("how many recent workflow runs to scan (max %d)", github.MaxFlakyRuns)) + flakyCmd.Flags().BoolVar(&flakyJSON, "json", false, "print the result as JSON") +} + +var flakyCmd = &cobra.Command{ + Use: "flaky ", + Short: "Find flaky checks: jobs that failed and then passed on re-run", + Long: `Scan a repository's recent GitHub Actions workflow runs for flaky jobs: jobs that +failed and then passed when the run was re-run, on the same commit. The code +didn't change between the attempts, so the failure wasn't caused by it. + +Only re-run workflow runs can show this. Jobs cancelled because another job failed +first are not counted. Results are grouped by workflow and job, most frequent first, +with a link to the latest failed attempt.`, + Example: ` prw flaky owner/repo + prw flaky https://github.com/owner/repo/pull/123 --runs 1000 + prw flaky owner/repo --json`, + SilenceUsage: true, + Args: func(cmd *cobra.Command, args []string) error { + if len(args) != 1 { + return waitErrorf(exitWaitError, "expected a repository (owner/repo) or a PR URL, got %d argument(s)", len(args)) + } + return nil + }, + RunE: func(cmd *cobra.Command, args []string) error { + owner, repo, err := parseRepoArg(args[0]) + if err != nil { + return waitErrorf(exitWaitError, "%v", err) + } + cfg, err := config.Load() + if err != nil { + return waitErrorf(exitWaitError, "failed to load config: %v", err) + } + token := cfg.GetToken() + if token == "" { + return &exitError{code: exitWaitError, err: errMissingToken} + } + + fmt.Fprintf(os.Stderr, "Scanning the last %d workflow runs of %s/%s...\n", flakyRuns, owner, repo) + report, err := github.FindFlaky(context.Background(), newGitHubClient(token), owner, repo, flakyRuns) + if err != nil { + return &exitError{code: exitWaitError, err: err} + } + for _, w := range report.Warnings { + fmt.Fprintf(os.Stderr, "Warning: %s\n", w) + } + if flakyJSON { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + if err := enc.Encode(report); err != nil { + return &exitError{code: exitWaitError, err: err} + } + return nil + } + printFlaky(os.Stdout, report) + return nil + }, +} + +var ( + repoSlug = regexp.MustCompile(`^([A-Za-z0-9_.-]+)/([A-Za-z0-9_.-]+)$`) + repoURL = regexp.MustCompile(`^(?:https?://)?github\.com/([^/]+)/([^/#?]+)`) +) + +// parseRepoArg accepts "owner/repo", a repository URL or a PR URL. +func parseRepoArg(arg string) (owner, repo string, err error) { + if m := repoSlug.FindStringSubmatch(arg); m != nil { + return m[1], m[2], nil + } + if m := repoURL.FindStringSubmatch(arg); m != nil { + return m[1], m[2], nil + } + return "", "", fmt.Errorf("invalid repository %q: use owner/repo or a GitHub URL", arg) +} + +func printFlaky(w io.Writer, r *github.FlakyReport) { + scope := fmt.Sprintf("%d workflow run(s)", r.Scanned) + if !r.Since.IsZero() { + scope += " since " + r.Since.Local().Format("2006-01-02") + } + fmt.Fprintf(w, "%s/%s: scanned %s, %d of them re-run.\n", r.Owner, r.Repo, scope, r.Reruns) + + switch { + case r.Reruns == 0: + fmt.Fprintln(w, "\nNo run was re-run, so there's nothing to compare: flakiness shows up when a failed\njob passes after re-running it on the same commit. Try a larger --runs.") + return + case len(r.Flaky) == 0: + fmt.Fprintln(w, "\nNo flaky checks: no job failed and then passed in a re-run.") + return + } + + fmt.Fprintln(w) + tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0) + fmt.Fprintln(tw, "FLAKED\tWORKFLOW\tJOB\tLAST SEEN\tLATEST FAILED ATTEMPT") + for _, f := range r.Flaky { + fmt.Fprintf(tw, "%dx\t%s\t%s\t%s\t%s\n", f.Count, f.Workflow, f.Job, f.LastSeen.Local().Format("2006-01-02 15:04"), f.FailedJobURL) + } + tw.Flush() +} diff --git a/cmd/prw/flaky_test.go b/cmd/prw/flaky_test.go new file mode 100644 index 0000000..491a42d --- /dev/null +++ b/cmd/prw/flaky_test.go @@ -0,0 +1,126 @@ +package main + +import ( + "bytes" + "fmt" + "net/http" + "strings" + "testing" + "time" + + "github.com/devblac/prw/internal/github" +) + +func TestParseRepoArg(t *testing.T) { + for arg, want := range map[string]string{ + "owner/repo": "owner/repo", + "devblac/prw.go": "devblac/prw.go", + "https://github.com/owner/repo": "owner/repo", + "https://github.com/owner/repo/pull/123": "owner/repo", + "github.com/owner/repo/actions/runs/1": "owner/repo", + "https://github.com/owner/repo#readme": "owner/repo", + } { + owner, repo, err := parseRepoArg(arg) + if err != nil || owner+"/"+repo != want { + t.Errorf("parseRepoArg(%q) = %s/%s, %v; want %s", arg, owner, repo, err, want) + } + } + for _, bad := range []string{"", "owner", "https://gitlab.com/o/r", "a/b/c"} { + if _, _, err := parseRepoArg(bad); err == nil { + t.Errorf("parseRepoArg(%q) should fail", bad) + } + } +} + +func flakyReport(flaky ...github.FlakyCheck) *github.FlakyReport { + return &github.FlakyReport{Owner: "o", Repo: "r", Scanned: 300, Reruns: 4, Since: time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC), Flaky: flaky} +} + +func TestPrintFlaky(t *testing.T) { + var buf bytes.Buffer + printFlaky(&buf, flakyReport(github.FlakyCheck{Workflow: "CI", Job: "Unit Tests", Count: 3, + LastSeen: time.Date(2026, 10, 24, 10, 0, 0, 0, time.UTC), FailedJobURL: "https://github.com/o/r/actions/runs/1/job/2"})) + out := buf.String() + for _, want := range []string{"o/r: scanned 300 workflow run(s) since 2026-10-", "4 of them re-run", "FLAKED", "3x", "Unit Tests", "https://github.com/o/r/actions/runs/1/job/2"} { + if !strings.Contains(out, want) { + t.Errorf("output lacks %q:\n%s", want, out) + } + } +} + +func TestPrintFlaky_NothingToCompare(t *testing.T) { + var buf bytes.Buffer + r := flakyReport() + r.Reruns = 0 + printFlaky(&buf, r) + if !strings.Contains(buf.String(), "No run was re-run") { + t.Errorf("output:\n%s", buf.String()) + } + buf.Reset() + printFlaky(&buf, flakyReport()) + if !strings.Contains(buf.String(), "No flaky checks") { + t.Errorf("output:\n%s", buf.String()) + } +} + +func TestFlakyCmd(t *testing.T) { + setupWaitCommand(t, func(w http.ResponseWriter, r *http.Request) { + switch { + case strings.HasSuffix(r.URL.Path, "/actions/runs"): + if r.URL.Query().Get("page") != "1" { + fmt.Fprint(w, `{"workflow_runs":[]}`) + return + } + fmt.Fprint(w, `{"workflow_runs":[{"id":9,"name":"CI","run_attempt":2,"created_at":"2026-10-24T10:00:00Z","html_url":"https://github.com/owner/repo/actions/runs/9"}]}`) + case strings.HasSuffix(r.URL.Path, "/runs/9/attempts/1/jobs"): + fmt.Fprint(w, `{"jobs":[{"name":"Test","conclusion":"failure","html_url":"https://github.com/owner/repo/actions/runs/9/job/1","completed_at":"2026-10-24T10:05:00Z"}]}`) + case strings.HasSuffix(r.URL.Path, "/runs/9/attempts/2/jobs"): + fmt.Fprint(w, `{"jobs":[{"name":"Test","conclusion":"success"}]}`) + default: + http.NotFound(w, r) + } + }) + flakyRuns, flakyJSON = 300, false + + out, err := captureStdout(func() error { + return flakyCmd.RunE(flakyCmd, []string{"https://github.com/owner/repo/pull/5"}) + }) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, "1x") || !strings.Contains(out, "Test") || !strings.Contains(out, "runs/9/job/1") { + t.Errorf("stdout:\n%s", out) + } +} + +func TestMCP_GetFlakyChecks(t *testing.T) { + c := successClient() + c.workflowRuns = []github.WorkflowRun{{ID: 9, Name: "CI", RunAttempt: 2}} + c.attemptJobs = map[string][]github.Job{ + "9/1": {{Name: "Test", Conclusion: "failure", HTMLURL: "https://github.com/o/r/actions/runs/9/job/1"}}, + "9/2": {{Name: "Test", Conclusion: "success"}}, + } + h := startMCP(t, staticClient(c)) + h.initialize() + + result, text := toolResultOf(t, h.call(2, "tools/call", map[string]interface{}{ + "name": "get_flaky_checks", "arguments": map[string]interface{}{"repo": "owner/repo"}, + })) + if result["isError"] == true { + t.Fatalf("tool error: %s", text) + } + if !strings.HasPrefix(text, "owner/repo: 1 recent workflow run(s) scanned, 1 re-run. Flaky: CI / Test (1x).") { + t.Errorf("text = %q", strings.SplitN(text, "\n", 2)[0]) + } + flaky := result["structuredContent"].(map[string]interface{})["flaky"].([]interface{}) + if len(flaky) != 1 || flaky[0].(map[string]interface{})["failed_job_url"] != "https://github.com/o/r/actions/runs/9/job/1" { + t.Errorf("flaky = %v", flaky) + } + + result, text = toolResultOf(t, h.call(3, "tools/call", map[string]interface{}{ + "name": "get_flaky_checks", "arguments": map[string]interface{}{"repo": "not a repo"}, + })) + if result["isError"] != true { + t.Errorf("an invalid repo must be a tool error: %s", text) + } +} diff --git a/cmd/prw/mcp.go b/cmd/prw/mcp.go index d111626..a88173a 100644 --- a/cmd/prw/mcp.go +++ b/cmd/prw/mcp.go @@ -80,6 +80,7 @@ Authentication is the same as for every other command (gh auth login, GITHUB_TOK type mcpClient interface { ciClient github.LogFetcher + github.FlakyFetcher } func defaultMCPClient() (mcpClient, error) { @@ -483,6 +484,29 @@ var mcpTools = []map[string]interface{}{ }, "annotations": map[string]interface{}{"readOnlyHint": true, "openWorldHint": true}, }, + { + "name": "get_flaky_checks", + "title": "Find flaky checks", + "description": "Find a repository's flaky GitHub Actions jobs: jobs that failed and then passed when the workflow run was re-run on the same commit, in its recent runs. Use it when a check fails to see whether that job is known to fail intermittently (a re-run may fix it) rather than because of the change. Returns workflow, job, how many re-runs it flaked in, when last, and the latest failed attempt's link. Scanning takes a few seconds on large repositories.", + "inputSchema": map[string]interface{}{ + "type": "object", + "properties": map[string]interface{}{ + "repo": map[string]interface{}{ + "type": "string", + "description": "The repository as owner/repo, or any GitHub URL in it (a PR URL works).", + }, + "runs": map[string]interface{}{ + "type": "integer", + "description": "How many recent workflow runs to scan (default 300, max 1000).", + "minimum": 1, + "maximum": github.MaxFlakyRuns, + }, + }, + "required": []string{"repo"}, + "additionalProperties": false, + }, + "annotations": map[string]interface{}{"readOnlyHint": true, "openWorldHint": true}, + }, { "name": "list_watched", "title": "List watched PRs", @@ -525,12 +549,13 @@ var mcpToolHandlers map[string]mcpToolHandler func init() { // Assigned in init to avoid an initialization cycle (handlers reference mcpServer). mcpToolHandlers = map[string]mcpToolHandler{ - "get_ci_status": toolGetCIStatus, - "wait_for_ci": toolWaitForCI, - "get_failed_logs": toolGetFailedLogs, - "list_watched": toolListWatched, - "watch_pr": toolWatchPR, - "unwatch_pr": toolUnwatchPR, + "get_ci_status": toolGetCIStatus, + "wait_for_ci": toolWaitForCI, + "get_failed_logs": toolGetFailedLogs, + "get_flaky_checks": toolGetFlakyChecks, + "list_watched": toolListWatched, + "watch_pr": toolWatchPR, + "unwatch_pr": toolUnwatchPR, } } @@ -688,6 +713,45 @@ func toolGetFailedLogs(ctx context.Context, s *mcpServer, args json.RawMessage, } } +func toolGetFlakyChecks(ctx context.Context, s *mcpServer, args json.RawMessage, _ io.Writer) mcpToolResult { + var opts struct { + Repo string `json:"repo"` + Runs int `json:"runs"` + } + if err := json.Unmarshal(args, &opts); err != nil { + return toolError("invalid arguments: %v", err) + } + owner, repo, err := parseRepoArg(opts.Repo) + if err != nil { + return toolError("%v", err) + } + client, err := s.githubClient() + if err != nil { + return toolError("%v", err) + } + report, err := github.FindFlaky(ctx, client, owner, repo, opts.Runs) + if err != nil { + return toolError("failed to scan %s/%s: %v", owner, repo, err) + } + return toolOK(flakySummary(report), report) +} + +// flakySummary is the one-line answer for an agent. +func flakySummary(r *github.FlakyReport) string { + scope := fmt.Sprintf("%s/%s: %d recent workflow run(s) scanned, %d re-run", r.Owner, r.Repo, r.Scanned, r.Reruns) + switch { + case r.Reruns == 0: + return scope + "; none was re-run, so flakiness can't be told apart from real failures." + case len(r.Flaky) == 0: + return scope + "; no job failed and then passed on re-run." + } + names := make([]string, 0, len(r.Flaky)) + for _, f := range r.Flaky { + names = append(names, fmt.Sprintf("%s / %s (%dx)", f.Workflow, f.Job, f.Count)) + } + return scope + ". Flaky: " + strings.Join(names, ", ") + "." +} + type watchedPROutput struct { URL string `json:"url"` Owner string `json:"owner"` diff --git a/cmd/prw/mcp_test.go b/cmd/prw/mcp_test.go index 45c49f5..35062cc 100644 --- a/cmd/prw/mcp_test.go +++ b/cmd/prw/mcp_test.go @@ -215,7 +215,7 @@ func TestMCP_ToolsList(t *testing.T) { t.Errorf("%s is listed but has no handler", name) } } - for _, want := range []string{"get_ci_status", "wait_for_ci", "get_failed_logs", "list_watched", "watch_pr", "unwatch_pr"} { + for _, want := range []string{"get_ci_status", "wait_for_ci", "get_failed_logs", "get_flaky_checks", "list_watched", "watch_pr", "unwatch_pr"} { if !names[want] { t.Errorf("tool %s missing from tools/list", want) } diff --git a/cmd/prw/wait_test.go b/cmd/prw/wait_test.go index 2c684c0..ba46fc3 100644 --- a/cmd/prw/wait_test.go +++ b/cmd/prw/wait_test.go @@ -50,6 +50,16 @@ type scriptedClient struct { current pollStep afterCI bool logs map[int64]string // job logs by check run id + workflowRuns []github.WorkflowRun + attemptJobs map[string][]github.Job // "runID/attempt" +} + +func (c *scriptedClient) ListWorkflowRuns(ctx context.Context, owner, repo string, max int) ([]github.WorkflowRun, error) { + return c.workflowRuns, nil +} + +func (c *scriptedClient) GetAttemptJobs(ctx context.Context, owner, repo string, runID int64, attempt int) ([]github.Job, error) { + return c.attemptJobs[fmt.Sprintf("%d/%d", runID, attempt)], nil } func (c *scriptedClient) GetPullRequest(ctx context.Context, owner, repo string, number int) (*github.PullRequest, error) { diff --git a/internal/github/flaky.go b/internal/github/flaky.go new file mode 100644 index 0000000..e5a393f --- /dev/null +++ b/internal/github/flaky.go @@ -0,0 +1,295 @@ +package github + +import ( + "context" + "fmt" + "sort" + "strings" + "sync" + "time" +) + +const ( + workflowRunsPerPage = 100 + // MaxFlakyRuns bounds how many recent workflow runs FindFlaky scans. + MaxFlakyRuns = 1000 + // DefaultFlakyRuns is how many recent workflow runs are scanned by default. + DefaultFlakyRuns = 300 + attemptJobsPerPage = 100 + maxAttemptJobPages = 5 +) + +// WorkflowRun is one run of a GitHub Actions workflow. +type WorkflowRun struct { + ID int64 `json:"id"` + Name string `json:"name"` // the workflow's name + HeadSHA string `json:"head_sha"` + RunAttempt int `json:"run_attempt"` + Status string `json:"status"` + Conclusion string `json:"conclusion"` + HTMLURL string `json:"html_url"` + CreatedAt time.Time `json:"created_at"` +} + +// Job is one job of one attempt of a workflow run. +type Job struct { + ID int64 `json:"id"` + Name string `json:"name"` + Status string `json:"status"` + Conclusion string `json:"conclusion"` + HTMLURL string `json:"html_url"` + CompletedAt time.Time `json:"completed_at"` +} + +// flakyConcurrency is how many requests FindFlaky has in flight at once. On large +// repositories GitHub takes seconds per runs/jobs request. +const flakyConcurrency = 4 + +// ListWorkflowRuns returns up to max of the repository's most recent workflow runs, +// newest first. Pages are fetched concurrently. +func (c *Client) ListWorkflowRuns(ctx context.Context, owner, repo string, max int) ([]WorkflowRun, error) { + pages := (max + workflowRunsPerPage - 1) / workflowRunsPerPage + results := make([][]WorkflowRun, pages) + err := forEachConcurrently(ctx, pages, func(i int) error { + path := fmt.Sprintf("/repos/%s/%s/actions/runs?per_page=%d&page=%d&exclude_pull_requests=true", + owner, repo, workflowRunsPerPage, i+1) + var resp struct { + WorkflowRuns []WorkflowRun `json:"workflow_runs"` + } + if err := c.getJSON(ctx, path, &resp); err != nil { + return err + } + results[i] = resp.WorkflowRuns + return nil + }) + if err != nil { + return nil, err + } + var runs []WorkflowRun + for _, page := range results { + runs = append(runs, page...) + if len(page) < workflowRunsPerPage { + break // the last page; later ones are empty + } + } + if len(runs) > max { + runs = runs[:max] + } + return runs, nil +} + +// forEachConcurrently calls fn(0..n-1) with at most flakyConcurrency in flight and +// returns the first error (by index). +func forEachConcurrently(ctx context.Context, n int, fn func(i int) error) error { + errs := make([]error, n) + sem := make(chan struct{}, flakyConcurrency) + var wg sync.WaitGroup + for i := 0; i < n; i++ { + select { + case sem <- struct{}{}: + case <-ctx.Done(): + wg.Wait() + return ctx.Err() + } + wg.Add(1) + go func(i int) { + defer wg.Done() + defer func() { <-sem }() + errs[i] = fn(i) + }(i) + } + wg.Wait() + for _, err := range errs { + if err != nil { + return err + } + } + return nil +} + +// GetAttemptJobs returns the jobs of one attempt of a workflow run. +func (c *Client) GetAttemptJobs(ctx context.Context, owner, repo string, runID int64, attempt int) ([]Job, error) { + var jobs []Job + for page := 1; page <= maxAttemptJobPages; page++ { + path := fmt.Sprintf("/repos/%s/%s/actions/runs/%d/attempts/%d/jobs?per_page=%d&page=%d", + owner, repo, runID, attempt, attemptJobsPerPage, page) + var resp struct { + Jobs []Job `json:"jobs"` + } + if err := c.getJSON(ctx, path, &resp); err != nil { + return nil, err + } + jobs = append(jobs, resp.Jobs...) + if len(resp.Jobs) < attemptJobsPerPage { + break + } + } + return jobs, nil +} + +// FlakyFetcher is what FindFlaky needs from the GitHub client. +type FlakyFetcher interface { + ListWorkflowRuns(ctx context.Context, owner, repo string, max int) ([]WorkflowRun, error) + GetAttemptJobs(ctx context.Context, owner, repo string, runID int64, attempt int) ([]Job, error) +} + +// FlakyCheck is a job that failed and then passed when re-run on the same commit. +type FlakyCheck struct { + Workflow string `json:"workflow"` + Job string `json:"job"` + // Count is how many re-run workflow runs it flaked in. + Count int `json:"count"` + LastSeen time.Time `json:"last_seen"` + // FailedJobURL is the most recent failed attempt, to look at its log. + FailedJobURL string `json:"failed_job_url"` +} + +// FlakyReport is the result of FindFlaky. +type FlakyReport struct { + Owner string `json:"owner"` + Repo string `json:"repo"` + // Scanned is how many workflow runs were looked at; Since is the oldest one. + Scanned int `json:"scanned_runs"` + Since time.Time `json:"since,omitempty"` + // Reruns is how many of them were re-run (only those can show flakiness). + Reruns int `json:"rerun_runs"` + Flaky []FlakyCheck `json:"flaky"` + // Warnings are runs whose jobs couldn't be fetched. + Warnings []string `json:"warnings,omitempty"` +} + +// FindFlaky scans the repository's last maxRuns workflow runs for jobs that failed +// and then passed in a later attempt of the same run: the same commit and workflow, +// so the failure wasn't caused by the code. Jobs cancelled because another job +// failed first are not counted. Results are grouped by workflow and job name, most +// frequent first. +func FindFlaky(ctx context.Context, c FlakyFetcher, owner, repo string, maxRuns int) (*FlakyReport, error) { + if maxRuns <= 0 { + maxRuns = DefaultFlakyRuns + } + if maxRuns > MaxFlakyRuns { + maxRuns = MaxFlakyRuns + } + runs, err := c.ListWorkflowRuns(ctx, owner, repo, maxRuns) + if err != nil { + return nil, fmt.Errorf("failed to list workflow runs: %w", err) + } + report := &FlakyReport{Owner: owner, Repo: repo, Scanned: len(runs), Flaky: []FlakyCheck{}} + if len(runs) > 0 { + report.Since = runs[len(runs)-1].CreatedAt + } + + // Fetch the jobs of every attempt of every re-run run, concurrently. + type fetch struct { + run int // index into reruns + attempt int + } + var reruns []WorkflowRun + var fetches []fetch + for _, run := range runs { + if run.RunAttempt < 2 { + continue + } + for a := 1; a <= run.RunAttempt; a++ { + fetches = append(fetches, fetch{len(reruns), a}) + } + reruns = append(reruns, run) + } + report.Reruns = len(reruns) + jobs := make([][][]Job, len(reruns)) // run → attempt → jobs + failed := make([]error, len(reruns)) + for i, run := range reruns { + jobs[i] = make([][]Job, run.RunAttempt) + } + var mu sync.Mutex + _ = forEachConcurrently(ctx, len(fetches), func(i int) error { + f := fetches[i] + got, err := c.GetAttemptJobs(ctx, owner, repo, reruns[f.run].ID, f.attempt) + mu.Lock() + defer mu.Unlock() + if err != nil && failed[f.run] == nil { + failed[f.run] = fmt.Errorf("attempt %d: %w", f.attempt, err) + } + jobs[f.run][f.attempt-1] = got + return nil // a failed run is reported as a warning, not an error + }) + if err := ctx.Err(); err != nil { + return nil, err + } + + type key struct{ workflow, job string } + found := map[key]*FlakyCheck{} + for i, run := range reruns { + if failed[i] != nil { + report.Warnings = append(report.Warnings, fmt.Sprintf("run %s %v", run.HTMLURL, failed[i])) + continue + } + for _, f := range flakyJobs(jobs[i]) { + k := key{run.Name, f.Name} + entry := found[k] + if entry == nil { + entry = &FlakyCheck{Workflow: run.Name, Job: f.Name} + found[k] = entry + } + entry.Count++ + seen := f.CompletedAt + if seen.IsZero() { + seen = run.CreatedAt + } + if entry.FailedJobURL == "" || seen.After(entry.LastSeen) { + entry.LastSeen, entry.FailedJobURL = seen, f.HTMLURL + } + } + } + + for _, f := range found { + report.Flaky = append(report.Flaky, *f) + } + sort.Slice(report.Flaky, func(i, j int) bool { + a, b := report.Flaky[i], report.Flaky[j] + if a.Count != b.Count { + return a.Count > b.Count + } + return a.LastSeen.After(b.LastSeen) + }) + return report, nil +} + +// flakyJobs returns, for one workflow run, the failed jobs (the latest failed +// attempt of each) whose name passed in a later attempt. +func flakyJobs(attempts [][]Job) []Job { + var flaky []Job + seen := map[string]bool{} + for a := len(attempts) - 2; a >= 0; a-- { // newest failure first + for _, job := range attempts[a] { + if seen[job.Name] || !jobFailed(job) { + continue + } + for _, later := range attempts[a+1:] { + if passed(later, job.Name) { + flaky = append(flaky, job) + seen[job.Name] = true + break + } + } + } + } + return flaky +} + +func jobFailed(j Job) bool { + switch strings.ToLower(j.Conclusion) { + case "failure", "timed_out": + return true + } + return false +} + +func passed(jobs []Job, name string) bool { + for _, j := range jobs { + if j.Name == name && strings.EqualFold(j.Conclusion, "success") { + return true + } + } + return false +} diff --git a/internal/github/flaky_test.go b/internal/github/flaky_test.go new file mode 100644 index 0000000..d0b35d0 --- /dev/null +++ b/internal/github/flaky_test.go @@ -0,0 +1,158 @@ +package github + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" +) + +type fakeFlaky struct { + runs []WorkflowRun + jobs map[string][]Job // "runID/attempt" + errs map[string]error + max int +} + +func (f *fakeFlaky) ListWorkflowRuns(ctx context.Context, owner, repo string, max int) ([]WorkflowRun, error) { + f.max = max + return f.runs, nil +} + +func (f *fakeFlaky) GetAttemptJobs(ctx context.Context, owner, repo string, runID int64, attempt int) ([]Job, error) { + k := fmt.Sprintf("%d/%d", runID, attempt) + if err := f.errs[k]; err != nil { + return nil, err + } + return f.jobs[k], nil +} + +var flakyNow = time.Date(2026, 10, 24, 12, 0, 0, 0, time.UTC) + +func wfRun(id int64, workflow string, attempts int, ago time.Duration) WorkflowRun { + return WorkflowRun{ID: id, Name: workflow, RunAttempt: attempts, CreatedAt: flakyNow.Add(-ago), + HTMLURL: fmt.Sprintf("https://github.com/o/r/actions/runs/%d", id)} +} + +func job(name, conclusion string, ago time.Duration) Job { + return Job{Name: name, Conclusion: conclusion, Status: "completed", CompletedAt: flakyNow.Add(-ago), + HTMLURL: "https://github.com/o/r/actions/runs/x/job/" + strings.ReplaceAll(name, " ", "_")} +} + +func TestFindFlaky(t *testing.T) { + f := &fakeFlaky{ + runs: []WorkflowRun{ + wfRun(1, "CI", 2, time.Hour), // Unit Tests failed, passed on re-run → flaky + wfRun(2, "CI", 1, 2*time.Hour), // single attempt: can't tell + wfRun(3, "CI", 3, 3*time.Hour), // Unit Tests flaked again (attempts 1 and 2) + wfRun(4, "Deploy", 2, 4*time.Hour), // Release failed twice: a real failure + }, + jobs: map[string][]Job{ + "1/1": {job("Unit Tests", "failure", 70*time.Minute), job("E2E", "cancelled", 70*time.Minute), job("Lint", "success", 70*time.Minute)}, + "1/2": {job("Unit Tests", "success", 60*time.Minute), job("E2E", "success", 60*time.Minute), job("Lint", "success", 70*time.Minute)}, + "3/1": {job("Unit Tests", "failure", 200*time.Minute)}, + "3/2": {job("Unit Tests", "timed_out", 190*time.Minute)}, + "3/3": {job("Unit Tests", "success", 180*time.Minute)}, + "4/1": {job("Release", "failure", 230*time.Minute)}, + "4/2": {job("Release", "failure", 220*time.Minute)}, + }, + } + + report, err := FindFlaky(context.Background(), f, "o", "r", 0) + if err != nil { + t.Fatal(err) + } + if f.max != DefaultFlakyRuns || report.Scanned != 4 || report.Reruns != 3 || !report.Since.Equal(flakyNow.Add(-4*time.Hour)) { + t.Errorf("report = %+v (max %d)", report, f.max) + } + if len(report.Flaky) != 1 { + t.Fatalf("flaky = %+v, want only Unit Tests (E2E was cancelled, Release never passed)", report.Flaky) + } + got := report.Flaky[0] + if got.Workflow != "CI" || got.Job != "Unit Tests" || got.Count != 2 || !got.LastSeen.Equal(flakyNow.Add(-70*time.Minute)) { + t.Errorf("flaky check = %+v, want CI / Unit Tests counted once per run, last seen in run 1", got) + } +} + +func TestFindFlaky_OrdersByCountThenRecency(t *testing.T) { + f := &fakeFlaky{ + runs: []WorkflowRun{wfRun(1, "CI", 2, time.Hour), wfRun(2, "CI", 2, 2*time.Hour), wfRun(3, "CI", 2, 3*time.Hour)}, + jobs: map[string][]Job{ + "1/1": {job("B", "failure", time.Hour)}, "1/2": {job("B", "success", time.Hour)}, + "2/1": {job("A", "failure", 2*time.Hour)}, "2/2": {job("A", "success", 2*time.Hour)}, + "3/1": {job("A", "failure", 3*time.Hour), job("C", "failure", 3*time.Hour)}, + "3/2": {job("A", "success", 3*time.Hour), job("C", "success", 3*time.Hour)}, + }, + } + report, _ := FindFlaky(context.Background(), f, "o", "r", 100) + var order []string + for _, c := range report.Flaky { + order = append(order, fmt.Sprintf("%s:%d", c.Job, c.Count)) + } + if strings.Join(order, " ") != "A:2 B:1 C:1" { + t.Errorf("order = %v", order) + } +} + +func TestFindFlaky_SkipsRunsWhoseJobsFail(t *testing.T) { + f := &fakeFlaky{ + runs: []WorkflowRun{wfRun(1, "CI", 2, time.Hour)}, + errs: map[string]error{"1/1": errors.New("boom")}, + } + report, err := FindFlaky(context.Background(), f, "o", "r", 100) + if err != nil { + t.Fatal(err) + } + if len(report.Flaky) != 0 || len(report.Warnings) != 1 || !strings.Contains(report.Warnings[0], "boom") { + t.Errorf("report = %+v", report) + } +} + +func TestFindFlaky_CapsRuns(t *testing.T) { + f := &fakeFlaky{} + FindFlaky(context.Background(), f, "o", "r", 5000) + if f.max != MaxFlakyRuns { + t.Errorf("max = %d, want %d", f.max, MaxFlakyRuns) + } +} + +func TestListWorkflowRuns_PagesInOrderAndStopsAtTheLastPage(t *testing.T) { + var requests int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt32(&requests, 1) + if r.URL.Query().Get("exclude_pull_requests") != "true" { + t.Error("runs must be listed without the pull_requests payload") + } + var page int + fmt.Sscan(r.URL.Query().Get("page"), &page) + count := map[int]int{1: 100, 2: 100, 3: 30}[page] // pages 4+ are empty + parts := make([]string, count) + for i := range parts { + parts[i] = fmt.Sprintf(`{"id":%d,"run_attempt":1}`, (page-1)*100+i+1) + } + fmt.Fprintf(w, `{"workflow_runs":[%s]}`, strings.Join(parts, ",")) + })) + defer srv.Close() + c := NewClient("t") + c.BaseURL = srv.URL + + runs, err := c.ListWorkflowRuns(context.Background(), "o", "r", 500) + if err != nil { + t.Fatal(err) + } + if len(runs) != 230 || runs[0].ID != 1 || runs[100].ID != 101 || runs[229].ID != 230 { + t.Errorf("got %d runs (first %d), want 230 in page order", len(runs), runs[0].ID) + } + if n := atomic.LoadInt32(&requests); n != 5 { + t.Errorf("%d requests, want 5 pages for max 500", n) + } + + if runs, _ := c.ListWorkflowRuns(context.Background(), "o", "r", 150); len(runs) != 150 { + t.Errorf("max 150: got %d runs", len(runs)) + } +}