feat(failure-intel): retry- and history-aware test status (#141)

CHANGELOG.md

@@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ### Unreleased
 
+- **`flakiness.historyDir` / `-Dseleniumboot.history.dir`** — metrics history can now live outside `target/`. Builds that clean `target/` or discard the CI workspace between runs had no history, so the Flakiness Radar and the newly/consistently-failing and intermittent statuses stayed empty. Point it at a directory your CI restores and saves (a cache or artifact store); the system property wins over the config. Default unchanged: `<reports dir>/metrics-history`. (#141)
+
+- **Retry- and history-aware test status** — each test now gets a status from this run's retries and the earlier runs in `target/metrics-history/`: **Passed on retry**, **Newly failing** (passed in every earlier run in the window), **Consistently failing** (failed in the two or more earlier runs) or **Intermittent** (mixed earlier results). A "Test status" card sits next to the Flakiness Radar, the status shows in the failure detail, and newly failing tests are listed first in the Failures tab. The metrics JSON gains `testStatus`. Radar thresholds and `failOnHighFlakiness` are unchanged. (#141)
+
 - **Failure grouping** — the report's Failures tab now opens with a "N tests, same cause" panel: failed tests are grouped by a signature of category, exception type, normalised message (ids, numbers and quoted values stripped) and top application stack frame. The flat per-class list is unchanged below it. With `ai.failureAnalysis` on, one Claude call is made per group; the other tests in the group reuse its analysis. The metrics JSON gains `failureSignature`. (#140)
 
 - **Failure classification** — every failed test is tagged with a category (`LOCATOR_NOT_FOUND`, `ELEMENT_NOT_INTERACTABLE`, `STALE_ELEMENT`, `TIMEOUT`, `ASSERTION_MISMATCH`, `NAVIGATION_ERROR`, `BROWSER_CRASH`, or `UNKNOWN`) and the evidence for it, from the exception class chain and message alone. No AI, no API key, no network. Shown as a badge in the HTML report's failure panel, as `failureCategory` / `failureEvidence` in the metrics JSON, and as the `type` of `<failure>` in the JUnit XML. The raw error and stack trace are unchanged. (#139)

docs-site/docs/changelog.md

@@ -11,6 +11,10 @@ All notable changes to Selenium Boot are documented here.
 
 ## [Unreleased]
 
+- **`flakiness.historyDir` / `-Dseleniumboot.history.dir`** — metrics history can now live outside `target/`. Builds that clean `target/` or discard the CI workspace between runs had no history, so the Flakiness Radar and the newly/consistently-failing and intermittent statuses stayed empty. Point it at a directory your CI restores and saves (a cache or artifact store); the system property wins over the config. Default unchanged: `<reports dir>/metrics-history`. (#141)
+
+- **Retry- and history-aware test status** — each test now gets a status from this run's retries and the earlier runs in `target/metrics-history/`: **Passed on retry**, **Newly failing** (passed in every earlier run in the window), **Consistently failing** (failed in the two or more earlier runs) or **Intermittent** (mixed earlier results). A "Test status" card sits next to the Flakiness Radar, the status shows in the failure detail, and newly failing tests are listed first in the Failures tab. The metrics JSON gains `testStatus`. Radar thresholds and `failOnHighFlakiness` are unchanged. (#141)
+
 - **Failure grouping** — the report's Failures tab now opens with a "N tests, same cause" panel: failed tests are grouped by a signature of category, exception type, normalised message (ids, numbers and quoted values stripped) and top application stack frame. The flat per-class list is unchanged below it. With `ai.failureAnalysis` on, one Claude call is made per group; the other tests in the group reuse its analysis. The metrics JSON gains `failureSignature`. (#140)
 
 - **`Locator` form actions: `clear()`, `check()`, `uncheck()`, `isChecked()`, `selectOption(text)`, `selectOptionByValue(value)`, `inputValue()`** — form controls can now be manipulated and inspected directly on `Locator` without falling back to `BasePage` helpers or raw `WebElement`s. `check()` and `uncheck()` are idempotent (no click if already in the target state), `isChecked()` queries checked state without waiting, and terminal actions (`clear`, `check`, `uncheck`, `selectOption`, `selectOptionByValue`, `inputValue`) auto-wait, re-resolving from scratch on every poll to survive stale or late-appearing elements. (#133)

docs-site/docs/reporting/html-report.md

@@ -84,6 +84,42 @@ When two or more failures share a cause, the Failures tab lists them above the f
 
 ---
 
+### Retry and history status
+
+Each test is also compared with its own earlier runs in `target/metrics-history/` (the last `flakiness.historyRuns`, the same window the Flakiness Radar uses). A "Test status" card next to the Radar lists the tests that need a second look, and the status appears at the top of the failure's detail panel:
+
+| Status | Meaning |
+|---|---|
+| **Newly failing** | Failed now; every earlier run in the window passed. Listed first in the Failures tab, since a recent change most likely broke it. |
+| **Consistently failing** | Failed now and in at least two earlier runs, with no pass among them. |
+| **Intermittent** | Failed now; the earlier runs contain both passes and failures. |
+| **Passed on retry** | Failed an attempt in this run, then passed on a retry. |
+
+A test that passed on its first attempt gets no status, and a test with no history gets none either (except passed on retry, which needs no history). Statuses are written to the metrics JSON as `testStatus`. The Radar's STABLE / WATCH / HIGH thresholds and the `failOnHighFlakiness` gate are unchanged.
+
+### Keeping history in CI
+
+The Flakiness Radar and every status except **Passed on retry** need earlier runs. If your build wipes `target/` (`mvn clean`) or the CI workspace is thrown away after each job, there is nothing to compare with and those parts of the report stay empty. Keep the history somewhere that survives, and point Selenium Boot at it:
+
+```yaml
+flakiness:
+  historyDir: .selenium-boot-history   # relative to the project, or an absolute path
+```
+
+or, without touching the config file, `-Dseleniumboot.history.dir=/path/to/history` (the system property wins).
+
+Then persist that directory between runs. With GitHub Actions, for example:
+
+```yaml
+- uses: actions/cache@v4
+  with:
+    path: .selenium-boot-history
+    key: selenium-boot-history-${{ github.run_id }}
+    restore-keys: selenium-boot-history-
+```
+
+On other CI systems, restore the directory from your artifact store before the run and upload it again afterwards. One JSON file is written per run, so the directory stays small; only the newest `flakiness.historyRuns` files are read.
+
 ## Self-contained format
 
 All screenshots are Base64-encoded and embedded inline. The report is a single file you can:

src/main/java/com/seleniumboot/config/SeleniumBootConfig.java

@@ -488,6 +488,7 @@ public static final class Flakiness {
         private int     historyRuns          = 20;
         private double  highRiskThreshold    = 33.0;
         private boolean failOnHighFlakiness  = false;
+        private String  historyDir           = "";
 
         public int     getHistoryRuns()                   { return historyRuns; }
         public void    setHistoryRuns(int v)              { this.historyRuns = v; }
@@ -497,6 +498,10 @@ public static final class Flakiness {
 
         public boolean isFailOnHighFlakiness()            { return failOnHighFlakiness; }
         public void    setFailOnHighFlakiness(boolean v)  { this.failOnHighFlakiness = v; }
+
+        /** Where metrics history is kept; blank means {@code <reports dir>/metrics-history}. */
+        public String  getHistoryDir()                    { return historyDir; }
+        public void    setHistoryDir(String v)            { this.historyDir = v == null ? "" : v; }
     }
 
     public static final class Tracing {

src/main/java/com/seleniumboot/flakiness/TestStatus.java

@@ -0,0 +1,23 @@
+package com.seleniumboot.flakiness;
+
+/** What this run's result means for a test, given its retries and recent history. */
+public enum TestStatus {
+    /** Failed an attempt in this run, then passed on a retry. */
+    PASSED_ON_RETRY("Passed on retry"),
+    /** Failed now after passing in every earlier run in the history window. */
+    NEWLY_FAILING("Newly failing"),
+    /** Failed now and in every earlier run in the history window (at least two of them). */
+    CONSISTENTLY_FAILING("Consistently failing"),
+    /** Both passes and failures in the earlier runs, so this result is not a surprise either way. */
+    INTERMITTENT("Intermittent");
+
+    private final String label;
+
+    TestStatus(String label) {
+        this.label = label;
+    }
+
+    public String label() {
+        return label;
+    }
+}

src/main/java/com/seleniumboot/flakiness/TestStatusAnalyzer.java

@@ -0,0 +1,98 @@
+package com.seleniumboot.flakiness;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.seleniumboot.config.SeleniumBootConfig;
+import com.seleniumboot.internal.SeleniumBootContext;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+/**
+ * Derives a per-test {@link TestStatus} from this run's result (including retries) and the
+ * earlier runs kept in {@code target/metrics-history/}.
+ *
+ * <p>It only labels tests that need a second look: a test that passed on its first attempt gets
+ * no status, however flaky its history (the Flakiness Radar covers that). The pass/fail ratio,
+ * the STABLE/WATCH/HIGH thresholds and the {@code failOnHighFlakiness} gate are unchanged.
+ */
+public final class TestStatusAnalyzer {
+
+    /** Prior failures needed, with no passes among them, before a failure is called consistent. */
+    static final int MIN_RUNS_FOR_CONSISTENT = 2;
+
+    private TestStatusAnalyzer() {}
+
+    /** One test's result in a single run. {@code status} is {@code PASSED}, {@code FAILED}, … */
+    public record Outcome(String status, int retryCount) {}
+
+    /**
+     * Pure classification.
+     *
+     * @param current  this run's result
+     * @param previous the test's earlier results, {@code PASSED} / {@code FAILED} only
+     *                 (skipped runs are not evidence either way), in any order
+     * @return the status, or {@code null} when there is nothing worth flagging or too little
+     *         history to say
+     */
+    public static TestStatus classify(Outcome current, List<String> previous) {
+        if (current == null) return null;
+        boolean passed = "PASSED".equals(current.status());
+        boolean failed = "FAILED".equals(current.status());
+        if (passed && current.retryCount() > 0) return TestStatus.PASSED_ON_RETRY;
+        if (!failed) return null;
+
+        int prevPass = 0;
+        int prevFail = 0;
+        for (String s : previous == null ? List.<String>of() : previous) {
+            if ("PASSED".equals(s)) prevPass++;
+            else if ("FAILED".equals(s)) prevFail++;
+        }
+        if (prevPass > 0 && prevFail > 0) return TestStatus.INTERMITTENT;
+        if (prevPass > 0) return TestStatus.NEWLY_FAILING;
+        if (prevFail >= MIN_RUNS_FOR_CONSISTENT) return TestStatus.CONSISTENTLY_FAILING;
+        return null;
+    }
+
+    /**
+     * Classifies every test in {@code current} against the history on disk. Never throws: with
+     * no readable history only {@link TestStatus#PASSED_ON_RETRY} can be decided.
+     *
+     * @param current       testId → this run's result
+     * @param excludeFiles  history file names written by this JVM, so a run is never compared
+     *                      with itself
+     */
+    public static Map<String, TestStatus> forRun(Map<String, Outcome> current, Set<String> excludeFiles) {
+        Map<String, List<String>> previous = new HashMap<>();
+        try {
+            SeleniumBootConfig.Flakiness cfg = config();
+            int historyRuns = cfg != null ? cfg.getHistoryRuns() : 20;
+            ObjectMapper mapper = new ObjectMapper();
+            for (File f : FlakinessAnalyzer.loadHistoryFiles(historyRuns)) {
+                if (excludeFiles != null && excludeFiles.contains(f.getName())) continue;
+                try {
+                    for (JsonNode t : mapper.readTree(f).path("tests")) {
+                        String id = t.path("testId").asText("");
+                        if (id.isEmpty()) continue;
+                        previous.computeIfAbsent(id, k -> new ArrayList<>()).add(t.path("status").asText(""));
+                    }
+                } catch (IOException ignored) {
+                    // unreadable history file: skip, as FlakinessAnalyzer does
+                }
+            }
+        } catch (RuntimeException ignored) {
+            // history is best-effort; fall through with whatever was read
+        }
+        Map<String, TestStatus> result = new LinkedHashMap<>();
+        for (Map.Entry<String, Outcome> e : current.entrySet()) {
+            TestStatus s = classify(e.getValue(), previous.get(e.getKey()));
+            if (s != null) result.put(e.getKey(), s);
+        }
+        return result;
+    }
+
+    private static SeleniumBootConfig.Flakiness config() {
+        try { return SeleniumBootContext.getConfig().getFlakiness(); } catch (Exception e) { return null; }
+    }
+}

src/main/java/com/seleniumboot/metrics/ExecutionMetrics.java

@@ -14,6 +14,9 @@ public final class ExecutionMetrics {
     private static final ConcurrentHashMap<String, Long> START_TIMES =
             new ConcurrentHashMap<>();
 
+    /** History copies written by this JVM; excluded when comparing against earlier runs. */
+    private static final Set<String> WRITTEN_HISTORY = ConcurrentHashMap.newKeySet();
+
     private static final ConcurrentHashMap<String, TestTiming> TIMINGS =
             new ConcurrentHashMap<>();
 
@@ -263,6 +266,20 @@ public static void printSummary() {
     // JSON Export
     // ==========================================================
 
+    /** Per-test status from this run's retries and the history on disk; empty if it can't be read. */
+    private static Map<String, com.seleniumboot.flakiness.TestStatus> testStatuses() {
+        try {
+            Map<String, com.seleniumboot.flakiness.TestStatusAnalyzer.Outcome> current = new LinkedHashMap<>();
+            for (TestTiming t : TIMINGS.values()) {
+                current.put(t.getTestId(), new com.seleniumboot.flakiness.TestStatusAnalyzer.Outcome(
+                        t.getStatus(), t.getRetryCount()));
+            }
+            return com.seleniumboot.flakiness.TestStatusAnalyzer.forRun(current, WRITTEN_HISTORY);
+        } catch (RuntimeException e) {
+            return Collections.emptyMap();
+        }
+    }
+
     public static void exportToJson() {
 
         Map<String, Object> report = new LinkedHashMap<>();
@@ -323,6 +340,8 @@ public static void exportToJson() {
 
         List<Map<String, Object>> testList = new ArrayList<>();
 
+        Map<String, com.seleniumboot.flakiness.TestStatus> testStatuses = testStatuses();
+
         for (TestTiming timing : TIMINGS.values()) {
 
             Map<String, Object> testEntry =
@@ -333,6 +352,10 @@ public static void exportToJson() {
             testEntry.put("thread",        timing.getThreadName());
             testEntry.put("status",        timing.getStatus() != null ? timing.getStatus() : "UNKNOWN");
             testEntry.put("retryCount",    timing.getRetryCount());
+            com.seleniumboot.flakiness.TestStatus testStatus = testStatuses.get(timing.getTestId());
+            if (testStatus != null) {
+                testEntry.put("testStatus", testStatus.name());
+            }
             testEntry.put("driverStartupMs", timing.getDriverStartupTime());
             testEntry.put("testLogicMs",   timing.getTestExecutionTime());
             testEntry.put("totalMs",       timing.getTotalTime());
@@ -415,10 +438,9 @@ public static void exportToJson() {
             if (!historyDir.exists()) {
                 historyDir.mkdirs();
             }
-            mapper.writeValue(
-                    new File(historyDir, "selenium-boot-metrics-" + timestamp + ".json"),
-                    report
-            );
+            String historyName = "selenium-boot-metrics-" + timestamp + ".json";
+            mapper.writeValue(new File(historyDir, historyName), report);
+            WRITTEN_HISTORY.add(historyName);
 
             System.out.println("[Selenium Boot] Metrics exported → " + primary.getPath());
             System.out.println("[Selenium Boot] History copy     → "
@@ -438,6 +460,7 @@ public static void exportToJson() {
     public static void reset() {
         START_TIMES.clear();
         TIMINGS.clear();
+        WRITTEN_HISTORY.clear();
         TOTAL_DURATION.set(0);
         SUITE_START.set(0);
     }