Skip to content

feat(failure-intel): retry- and history-aware test status (#141) - #152

Merged
shossain786 merged 2 commits into
masterfrom
feat/retry-history-aware-status
Oct 13, 2026
Merged

shossain786 merged 2 commits into
masterfrom
feat/retry-history-aware-status

Conversation

@shossain786

Copy link
Copy Markdown
Contributor

What & why

Flakiness was a pass/fail ratio only and ignored retries. Each test now gets a status from this run's retries and the earlier runs in target/metrics-history/:

Status Rule
Passed on retry failed an attempt, then passed
Newly failing failed now; every earlier run in the window passed
Consistently failing failed now and in at least 2 earlier runs, no pass among them
Intermittent failed now; earlier runs mixed

A test that passed first time, or has no history, gets no status (except passed on retry).

  • TestStatusAnalyzer.classify(...) is a pure function; forRun(...) reads the history.
  • Computed in exportToJson() before this run's history copy is written, and files written by this JVM are excluded, so a run is never compared with itself (TestNG fires onFinish per suite).
  • Report: "Test status" card next to the Flakiness Radar, badge in the failure detail, newly failing listed first in the Failures tab.
  • Metrics JSON gains testStatus.
  • No change to STABLE/WATCH/HIGH thresholds or failOnHighFlakiness.

Part of #138. Closes #141.

Verification

mvn test: 659 run, 0 failures (638 on master + 21 new): empty history, single run, skipped runs, unreadable history file, double export in one JVM, report ordering.

Notes

🤖 Generated with Claude Code

shossain786 and others added 2 commits October 13, 2026 07:29
Each test gets a status from this run's retries and the earlier runs in
target/metrics-history/: passed on retry, newly failing, consistently
failing, intermittent. A pure classify() function, wired into the metrics
export (testStatus), shown in a card next to the Flakiness Radar and as a
badge in the failure detail. Newly failing tests are listed first in the
Failures tab. Radar thresholds and failOnHighFlakiness are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…get (#141)

Builds that clean target/ or discard the CI workspace between runs had no
metrics history, so the Flakiness Radar and the newly/consistently-failing
and intermittent statuses were always empty. History can now live anywhere:
-Dseleniumboot.history.dir, then flakiness.historyDir, then the existing
<reports dir>/metrics-history default. Documented with a CI cache recipe.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@shossain786

Copy link
Copy Markdown
Contributor Author

Added flakiness.historyDir / -Dseleniumboot.history.dir to this PR: history can now live outside target/, so builds that clean it or discard the CI workspace can still get the Radar and the statuses. Default location unchanged; docs have a GitHub Actions cache recipe. mvn test 663/0.

🤖 Generated with Claude Code

@shossain786
shossain786 merged commit c89f55b into master Oct 13, 2026
1 check passed
@shossain786
shossain786 deleted the feat/retry-history-aware-status branch October 13, 2026 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(failure-intel): retry- and history-aware failure status

1 participant