Skip to content

fix(ci): Improve flaky test issue deduplication#22042

Open
mydea wants to merge 1 commit into
developfrom
feat/dedupe-flaky-ci-issues
Open

fix(ci): Improve flaky test issue deduplication#22042
mydea wants to merge 1 commit into
developfrom
feat/dedupe-flaky-ci-issues

Conversation

@mydea

@mydea mydea commented Jul 8, 2026

Copy link
Copy Markdown
Member

report-ci-failures.mjs opens one GitHub issue per failing test, keyed on a normalized job + test name. Several variant patterns weren't being stripped, so the same flaky test kept opening duplicate issues. This widens both normalizers.

normalizeJobName now also collapses:

  • (Node 22) and (n/m) shard groups (in addition to the existing (22) / (TS 3.8))
  • All Playwright bundle/esm build configs into a single Playwright Tests bucket (e.g. bundle_tracing, bundle_tracing_replay, esm (1/4)) — the same profiling flake was filing an issue per config
  • E2E -node-20-18 app-name version suffixes

Non-version parentheticals such as (Orchestrion) are preserved.

normalizeTestName now also strips:

  • The [chromium] › browser project prefix
  • Playwright test.ts:33:11 line/column locations — these drift whenever the test file is edited, silently reopening the same flaky issue under a new title
  • The trailing [esm] / [cjs] module-format suffix (the main gap — the old describe-block form was handled but the newer suffix form was not) and the bare esm/cjs > describe segment, so both forms converge

Verified against current open Flaky Test issues: the [esm]/[cjs] postgres/mysql pairs collapse to one, the profiling test across bundle_tracing / bundle_tracing_replay / esm (1/4) collapses to one, and line-drift variants merge.

Note: dedup still matches on exact issue title, so existing issues (which carry the old normalized titles) won't retroactively merge — the next run files them once under the new canonical titles. Bulk-closing the current flaky issues after merge is recommended.

🤖 Generated with Claude Code

Strip more matrix/variant noise from job and test names so the same flaky
test dedupes to a single issue instead of one per variant.

Job names now also collapse (Node xx) and (n/m) shard groups, fold all
Playwright bundle/esm build configs into one bucket, and drop E2E
"-node-xx" app-name suffixes.

Test names now also strip the [chromium] browser prefix, Playwright
line:column locations (which drift on every file edit and reopened
issues), the trailing [esm]/[cjs] module-format suffix, and the bare
"esm/cjs >" describe segment so the suffix and describe-block forms
converge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6d78642. Configure here.

.replace(/(\.[cm]?[jt]sx?):\d+:\d+/gi, '$1')
.replace(/esm\/cjs\s*>\s*(?:esm|cjs)\b/gi, 'esm/cjs')
.replace(/\besm\/cjs\s*>\s*/gi, '')
.replace(/\s*\[(?:esm|cjs)\]/gi, '')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing normalization regression tests

Low Severity

This fix expands normalizeJobName and normalizeTestName, but the PR adds no unit tests for those cases. Without table-driven examples (Playwright job folding, [esm]/[cjs] titles, line-number stripping), future regex edits can silently break flaky-issue deduplication again.

Fix in Cursor Fix in Web

Triggered by project rule: PR Review Guidelines for Cursor Bot

Reviewed by Cursor Bugbot for commit 6d78642. Configure here.

@mydea mydea requested review from chargome and nicohrubec July 9, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant