diff --git a/CHANGELOG.md b/CHANGELOG.md index 4147da7..b4e553b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Unreleased +- **`flakiness.historyDir` / `-Dseleniumboot.history.dir`** — metrics history can now live outside `target/`. Builds that clean `target/` or discard the CI workspace between runs had no history, so the Flakiness Radar and the newly/consistently-failing and intermittent statuses stayed empty. Point it at a directory your CI restores and saves (a cache or artifact store); the system property wins over the config. Default unchanged: `/metrics-history`. (#141) + +- **Retry- and history-aware test status** — each test now gets a status from this run's retries and the earlier runs in `target/metrics-history/`: **Passed on retry**, **Newly failing** (passed in every earlier run in the window), **Consistently failing** (failed in the two or more earlier runs) or **Intermittent** (mixed earlier results). A "Test status" card sits next to the Flakiness Radar, the status shows in the failure detail, and newly failing tests are listed first in the Failures tab. The metrics JSON gains `testStatus`. Radar thresholds and `failOnHighFlakiness` are unchanged. (#141) + - **Failure grouping** — the report's Failures tab now opens with a "N tests, same cause" panel: failed tests are grouped by a signature of category, exception type, normalised message (ids, numbers and quoted values stripped) and top application stack frame. The flat per-class list is unchanged below it. With `ai.failureAnalysis` on, one Claude call is made per group; the other tests in the group reuse its analysis. The metrics JSON gains `failureSignature`. (#140) - **Failure classification** — every failed test is tagged with a category (`LOCATOR_NOT_FOUND`, `ELEMENT_NOT_INTERACTABLE`, `STALE_ELEMENT`, `TIMEOUT`, `ASSERTION_MISMATCH`, `NAVIGATION_ERROR`, `BROWSER_CRASH`, or `UNKNOWN`) and the evidence for it, from the exception class chain and message alone. No AI, no API key, no network. Shown as a badge in the HTML report's failure panel, as `failureCategory` / `failureEvidence` in the metrics JSON, and as the `type` of `` in the JUnit XML. The raw error and stack trace are unchanged. (#139) diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index f940300..97172f3 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -11,6 +11,10 @@ All notable changes to Selenium Boot are documented here. ## [Unreleased] +- **`flakiness.historyDir` / `-Dseleniumboot.history.dir`** — metrics history can now live outside `target/`. Builds that clean `target/` or discard the CI workspace between runs had no history, so the Flakiness Radar and the newly/consistently-failing and intermittent statuses stayed empty. Point it at a directory your CI restores and saves (a cache or artifact store); the system property wins over the config. Default unchanged: `/metrics-history`. (#141) + +- **Retry- and history-aware test status** — each test now gets a status from this run's retries and the earlier runs in `target/metrics-history/`: **Passed on retry**, **Newly failing** (passed in every earlier run in the window), **Consistently failing** (failed in the two or more earlier runs) or **Intermittent** (mixed earlier results). A "Test status" card sits next to the Flakiness Radar, the status shows in the failure detail, and newly failing tests are listed first in the Failures tab. The metrics JSON gains `testStatus`. Radar thresholds and `failOnHighFlakiness` are unchanged. (#141) + - **Failure grouping** — the report's Failures tab now opens with a "N tests, same cause" panel: failed tests are grouped by a signature of category, exception type, normalised message (ids, numbers and quoted values stripped) and top application stack frame. The flat per-class list is unchanged below it. With `ai.failureAnalysis` on, one Claude call is made per group; the other tests in the group reuse its analysis. The metrics JSON gains `failureSignature`. (#140) - **`Locator` form actions: `clear()`, `check()`, `uncheck()`, `isChecked()`, `selectOption(text)`, `selectOptionByValue(value)`, `inputValue()`** — form controls can now be manipulated and inspected directly on `Locator` without falling back to `BasePage` helpers or raw `WebElement`s. `check()` and `uncheck()` are idempotent (no click if already in the target state), `isChecked()` queries checked state without waiting, and terminal actions (`clear`, `check`, `uncheck`, `selectOption`, `selectOptionByValue`, `inputValue`) auto-wait, re-resolving from scratch on every poll to survive stale or late-appearing elements. (#133) diff --git a/docs-site/docs/reporting/html-report.md b/docs-site/docs/reporting/html-report.md index 509c7d9..f29b876 100644 --- a/docs-site/docs/reporting/html-report.md +++ b/docs-site/docs/reporting/html-report.md @@ -84,6 +84,42 @@ When two or more failures share a cause, the Failures tab lists them above the f --- +### Retry and history status + +Each test is also compared with its own earlier runs in `target/metrics-history/` (the last `flakiness.historyRuns`, the same window the Flakiness Radar uses). A "Test status" card next to the Radar lists the tests that need a second look, and the status appears at the top of the failure's detail panel: + +| Status | Meaning | +|---|---| +| **Newly failing** | Failed now; every earlier run in the window passed. Listed first in the Failures tab, since a recent change most likely broke it. | +| **Consistently failing** | Failed now and in at least two earlier runs, with no pass among them. | +| **Intermittent** | Failed now; the earlier runs contain both passes and failures. | +| **Passed on retry** | Failed an attempt in this run, then passed on a retry. | + +A test that passed on its first attempt gets no status, and a test with no history gets none either (except passed on retry, which needs no history). Statuses are written to the metrics JSON as `testStatus`. The Radar's STABLE / WATCH / HIGH thresholds and the `failOnHighFlakiness` gate are unchanged. + +### Keeping history in CI + +The Flakiness Radar and every status except **Passed on retry** need earlier runs. If your build wipes `target/` (`mvn clean`) or the CI workspace is thrown away after each job, there is nothing to compare with and those parts of the report stay empty. Keep the history somewhere that survives, and point Selenium Boot at it: + +```yaml +flakiness: + historyDir: .selenium-boot-history # relative to the project, or an absolute path +``` + +or, without touching the config file, `-Dseleniumboot.history.dir=/path/to/history` (the system property wins). + +Then persist that directory between runs. With GitHub Actions, for example: + +```yaml +- uses: actions/cache@v4 + with: + path: .selenium-boot-history + key: selenium-boot-history-${{ github.run_id }} + restore-keys: selenium-boot-history- +``` + +On other CI systems, restore the directory from your artifact store before the run and upload it again afterwards. One JSON file is written per run, so the directory stays small; only the newest `flakiness.historyRuns` files are read. + ## Self-contained format All screenshots are Base64-encoded and embedded inline. The report is a single file you can: diff --git a/src/main/java/com/seleniumboot/config/SeleniumBootConfig.java b/src/main/java/com/seleniumboot/config/SeleniumBootConfig.java index 17fa52e..3fc599e 100644 --- a/src/main/java/com/seleniumboot/config/SeleniumBootConfig.java +++ b/src/main/java/com/seleniumboot/config/SeleniumBootConfig.java @@ -488,6 +488,7 @@ public static final class Flakiness { private int historyRuns = 20; private double highRiskThreshold = 33.0; private boolean failOnHighFlakiness = false; + private String historyDir = ""; public int getHistoryRuns() { return historyRuns; } public void setHistoryRuns(int v) { this.historyRuns = v; } @@ -497,6 +498,10 @@ public static final class Flakiness { public boolean isFailOnHighFlakiness() { return failOnHighFlakiness; } public void setFailOnHighFlakiness(boolean v) { this.failOnHighFlakiness = v; } + + /** Where metrics history is kept; blank means {@code /metrics-history}. */ + public String getHistoryDir() { return historyDir; } + public void setHistoryDir(String v) { this.historyDir = v == null ? "" : v; } } public static final class Tracing { diff --git a/src/main/java/com/seleniumboot/flakiness/TestStatus.java b/src/main/java/com/seleniumboot/flakiness/TestStatus.java new file mode 100644 index 0000000..6ed86b7 --- /dev/null +++ b/src/main/java/com/seleniumboot/flakiness/TestStatus.java @@ -0,0 +1,23 @@ +package com.seleniumboot.flakiness; + +/** What this run's result means for a test, given its retries and recent history. */ +public enum TestStatus { + /** Failed an attempt in this run, then passed on a retry. */ + PASSED_ON_RETRY("Passed on retry"), + /** Failed now after passing in every earlier run in the history window. */ + NEWLY_FAILING("Newly failing"), + /** Failed now and in every earlier run in the history window (at least two of them). */ + CONSISTENTLY_FAILING("Consistently failing"), + /** Both passes and failures in the earlier runs, so this result is not a surprise either way. */ + INTERMITTENT("Intermittent"); + + private final String label; + + TestStatus(String label) { + this.label = label; + } + + public String label() { + return label; + } +} diff --git a/src/main/java/com/seleniumboot/flakiness/TestStatusAnalyzer.java b/src/main/java/com/seleniumboot/flakiness/TestStatusAnalyzer.java new file mode 100644 index 0000000..38b07d9 --- /dev/null +++ b/src/main/java/com/seleniumboot/flakiness/TestStatusAnalyzer.java @@ -0,0 +1,98 @@ +package com.seleniumboot.flakiness; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.seleniumboot.config.SeleniumBootConfig; +import com.seleniumboot.internal.SeleniumBootContext; + +import java.io.File; +import java.io.IOException; +import java.util.*; + +/** + * Derives a per-test {@link TestStatus} from this run's result (including retries) and the + * earlier runs kept in {@code target/metrics-history/}. + * + *

It only labels tests that need a second look: a test that passed on its first attempt gets + * no status, however flaky its history (the Flakiness Radar covers that). The pass/fail ratio, + * the STABLE/WATCH/HIGH thresholds and the {@code failOnHighFlakiness} gate are unchanged. + */ +public final class TestStatusAnalyzer { + + /** Prior failures needed, with no passes among them, before a failure is called consistent. */ + static final int MIN_RUNS_FOR_CONSISTENT = 2; + + private TestStatusAnalyzer() {} + + /** One test's result in a single run. {@code status} is {@code PASSED}, {@code FAILED}, … */ + public record Outcome(String status, int retryCount) {} + + /** + * Pure classification. + * + * @param current this run's result + * @param previous the test's earlier results, {@code PASSED} / {@code FAILED} only + * (skipped runs are not evidence either way), in any order + * @return the status, or {@code null} when there is nothing worth flagging or too little + * history to say + */ + public static TestStatus classify(Outcome current, List previous) { + if (current == null) return null; + boolean passed = "PASSED".equals(current.status()); + boolean failed = "FAILED".equals(current.status()); + if (passed && current.retryCount() > 0) return TestStatus.PASSED_ON_RETRY; + if (!failed) return null; + + int prevPass = 0; + int prevFail = 0; + for (String s : previous == null ? List.of() : previous) { + if ("PASSED".equals(s)) prevPass++; + else if ("FAILED".equals(s)) prevFail++; + } + if (prevPass > 0 && prevFail > 0) return TestStatus.INTERMITTENT; + if (prevPass > 0) return TestStatus.NEWLY_FAILING; + if (prevFail >= MIN_RUNS_FOR_CONSISTENT) return TestStatus.CONSISTENTLY_FAILING; + return null; + } + + /** + * Classifies every test in {@code current} against the history on disk. Never throws: with + * no readable history only {@link TestStatus#PASSED_ON_RETRY} can be decided. + * + * @param current testId → this run's result + * @param excludeFiles history file names written by this JVM, so a run is never compared + * with itself + */ + public static Map forRun(Map current, Set excludeFiles) { + Map> previous = new HashMap<>(); + try { + SeleniumBootConfig.Flakiness cfg = config(); + int historyRuns = cfg != null ? cfg.getHistoryRuns() : 20; + ObjectMapper mapper = new ObjectMapper(); + for (File f : FlakinessAnalyzer.loadHistoryFiles(historyRuns)) { + if (excludeFiles != null && excludeFiles.contains(f.getName())) continue; + try { + for (JsonNode t : mapper.readTree(f).path("tests")) { + String id = t.path("testId").asText(""); + if (id.isEmpty()) continue; + previous.computeIfAbsent(id, k -> new ArrayList<>()).add(t.path("status").asText("")); + } + } catch (IOException ignored) { + // unreadable history file: skip, as FlakinessAnalyzer does + } + } + } catch (RuntimeException ignored) { + // history is best-effort; fall through with whatever was read + } + Map result = new LinkedHashMap<>(); + for (Map.Entry e : current.entrySet()) { + TestStatus s = classify(e.getValue(), previous.get(e.getKey())); + if (s != null) result.put(e.getKey(), s); + } + return result; + } + + private static SeleniumBootConfig.Flakiness config() { + try { return SeleniumBootContext.getConfig().getFlakiness(); } catch (Exception e) { return null; } + } +} diff --git a/src/main/java/com/seleniumboot/metrics/ExecutionMetrics.java b/src/main/java/com/seleniumboot/metrics/ExecutionMetrics.java index b0831f1..65a2336 100644 --- a/src/main/java/com/seleniumboot/metrics/ExecutionMetrics.java +++ b/src/main/java/com/seleniumboot/metrics/ExecutionMetrics.java @@ -14,6 +14,9 @@ public final class ExecutionMetrics { private static final ConcurrentHashMap START_TIMES = new ConcurrentHashMap<>(); + /** History copies written by this JVM; excluded when comparing against earlier runs. */ + private static final Set WRITTEN_HISTORY = ConcurrentHashMap.newKeySet(); + private static final ConcurrentHashMap TIMINGS = new ConcurrentHashMap<>(); @@ -263,6 +266,20 @@ public static void printSummary() { // JSON Export // ========================================================== + /** Per-test status from this run's retries and the history on disk; empty if it can't be read. */ + private static Map testStatuses() { + try { + Map current = new LinkedHashMap<>(); + for (TestTiming t : TIMINGS.values()) { + current.put(t.getTestId(), new com.seleniumboot.flakiness.TestStatusAnalyzer.Outcome( + t.getStatus(), t.getRetryCount())); + } + return com.seleniumboot.flakiness.TestStatusAnalyzer.forRun(current, WRITTEN_HISTORY); + } catch (RuntimeException e) { + return Collections.emptyMap(); + } + } + public static void exportToJson() { Map report = new LinkedHashMap<>(); @@ -323,6 +340,8 @@ public static void exportToJson() { List> testList = new ArrayList<>(); + Map testStatuses = testStatuses(); + for (TestTiming timing : TIMINGS.values()) { Map testEntry = @@ -333,6 +352,10 @@ public static void exportToJson() { testEntry.put("thread", timing.getThreadName()); testEntry.put("status", timing.getStatus() != null ? timing.getStatus() : "UNKNOWN"); testEntry.put("retryCount", timing.getRetryCount()); + com.seleniumboot.flakiness.TestStatus testStatus = testStatuses.get(timing.getTestId()); + if (testStatus != null) { + testEntry.put("testStatus", testStatus.name()); + } testEntry.put("driverStartupMs", timing.getDriverStartupTime()); testEntry.put("testLogicMs", timing.getTestExecutionTime()); testEntry.put("totalMs", timing.getTotalTime()); @@ -415,10 +438,9 @@ public static void exportToJson() { if (!historyDir.exists()) { historyDir.mkdirs(); } - mapper.writeValue( - new File(historyDir, "selenium-boot-metrics-" + timestamp + ".json"), - report - ); + String historyName = "selenium-boot-metrics-" + timestamp + ".json"; + mapper.writeValue(new File(historyDir, historyName), report); + WRITTEN_HISTORY.add(historyName); System.out.println("[Selenium Boot] Metrics exported → " + primary.getPath()); System.out.println("[Selenium Boot] History copy → " @@ -438,6 +460,7 @@ public static void exportToJson() { public static void reset() { START_TIMES.clear(); TIMINGS.clear(); + WRITTEN_HISTORY.clear(); TOTAL_DURATION.set(0); SUITE_START.set(0); } diff --git a/src/main/java/com/seleniumboot/reporting/HtmlReportGenerator.java b/src/main/java/com/seleniumboot/reporting/HtmlReportGenerator.java index 8a4277b..d9eaac9 100644 --- a/src/main/java/com/seleniumboot/reporting/HtmlReportGenerator.java +++ b/src/main/java/com/seleniumboot/reporting/HtmlReportGenerator.java @@ -232,7 +232,7 @@ static String buildHtml(JsonNode root) { String retrySection = buildRetrySection(flakyTests, recoveredTests); String slowestSection = tests != null ? buildSlowestTests(tests) : ""; - String flakinessSection = buildFlakinessSection(); + String flakinessSection = buildFlakinessSection() + buildTestStatusSection(tests); String failureRows = buildFailureRows(tests, isMatrixRun); String failureGroups = buildFailureGroups(tests); String failureBadge = failedTests > 0 @@ -339,10 +339,11 @@ private static String buildRow(JsonNode test, int rowIndex, String groupId, bool String stepsHtml = buildStepTimeline(test); boolean hasDetail = errorMsg != null || stackTrace != null || test.has("failureCategory") || !stepsHtml.isEmpty() - || recordingPath != null || aiAnalysis != null || perfNode != null; + || recordingPath != null || aiAnalysis != null || perfNode != null || test.has("testStatus"); String detailRow = ""; if (hasDetail) { - String whyHtml = test.has("failureCategory") ? buildFailureWhy(test) : ""; + String whyHtml = buildTestStatusBadge(test) + + (test.has("failureCategory") ? buildFailureWhy(test) : ""); String errorHtml = whyHtml + (errorMsg != null ? "

" + escapeHtml(errorMsg) + "
" : ""); String traceHtml = stackTrace != null ? "
"    + escapeHtml(stackTrace) + "
" : ""; String recordingHtml = recordingPath != null ? buildRecordingCell(recordingPath) : ""; @@ -400,14 +401,71 @@ private static String buildFailureWhy(JsonNode test) { + escapeHtml(evidence) + ""; } + private static String testStatusOf(JsonNode test) { + return test.has("testStatus") ? test.get("testStatus").asText() : ""; + } + + private static String testStatusLabel(String status) { + try { + return com.seleniumboot.flakiness.TestStatus.valueOf(status).label(); + } catch (IllegalArgumentException e) { + return status; + } + } + + private static String buildTestStatusBadge(JsonNode test) { + String status = testStatusOf(test); + if (status.isEmpty()) return ""; + return "
" + + escapeHtml(testStatusLabel(status)) + "
"; + } + + /** Tests flagged by retries or history, next to the Flakiness Radar. Empty when none are. */ + private static String buildTestStatusSection(JsonNode tests) { + if (tests == null) return ""; + java.util.List flagged = new java.util.ArrayList<>(); + for (JsonNode t : tests) { + if (!testStatusOf(t).isEmpty()) flagged.add(t); + } + if (flagged.isEmpty()) return ""; + String[] order = {"NEWLY_FAILING", "CONSISTENTLY_FAILING", "INTERMITTENT", "PASSED_ON_RETRY"}; + flagged.sort(java.util.Comparator.comparingInt(t -> java.util.Arrays.asList(order).indexOf(testStatusOf(t)))); + + StringBuilder sb = new StringBuilder(); + sb.append("
\n"); + sb.append("
⚠ Test status (this run vs history)
\n"); + sb.append("
\n"); + for (JsonNode t : flagged) { + String status = testStatusOf(t); + String cls = "NEWLY_FAILING".equals(status) || "CONSISTENTLY_FAILING".equals(status) + ? "risk-high" : "risk-watch"; + String id = t.has("testId") ? t.get("testId").asText() : ""; + int lastDot = id.lastIndexOf('.'); + String shortName = lastDot >= 0 ? id.substring(lastDot + 1) : id; + sb.append("
\n"); + sb.append(" ") + .append(escapeHtml(testStatusLabel(status))).append("\n"); + sb.append(" ") + .append(escapeHtml(shortName)).append("\n"); + sb.append("
\n"); + } + sb.append("
\n
\n"); + return sb.toString(); + } + private static String buildFailureRows(JsonNode tests) { return buildFailureRows(tests, false); } private static String buildFailureRows(JsonNode tests, boolean showBrowser) { if (tests == null) return noFailuresRow(); + // Newly failing first: it is the one a recent change most likely broke. The sort is + // stable, so the classes and tests keep their order otherwise. + java.util.List ordered = new java.util.ArrayList<>(); + tests.forEach(ordered::add); + ordered.sort(java.util.Comparator.comparingInt(t -> "NEWLY_FAILING".equals(testStatusOf(t)) ? 0 : 1)); java.util.Map> byClass = new java.util.LinkedHashMap<>(); - for (JsonNode test : tests) { + for (JsonNode test : ordered) { if (!"FAILED".equals(test.has("status") ? test.get("status").asText() : "")) continue; String cls = test.has("testClassName") ? test.get("testClassName").asText() : ""; if (cls.isEmpty()) cls = "Unknown"; diff --git a/src/main/java/com/seleniumboot/reporting/ReportPaths.java b/src/main/java/com/seleniumboot/reporting/ReportPaths.java index edf9e0e..4458118 100644 --- a/src/main/java/com/seleniumboot/reporting/ReportPaths.java +++ b/src/main/java/com/seleniumboot/reporting/ReportPaths.java @@ -69,8 +69,29 @@ public static File htmlReport() { return new File(baseDir(), "selenium-boot-report.html"); } - /** {@code /metrics-history} — timestamped metrics copies for flakiness analysis. */ + /** + * Where timestamped metrics copies are kept for flakiness and test-status analysis. + * Resolution order: the {@code seleniumboot.history.dir} system property, then + * {@code flakiness.historyDir} in the config, then {@code /metrics-history}. + * + *

Point it outside {@code target/} (or restore it before the run) when the build + * directory is cleaned or the CI workspace is discarded between runs, otherwise there is + * no history to compare against. + */ public static File metricsHistoryDir() { + String override = System.getProperty("seleniumboot.history.dir"); + if (override != null && !override.isBlank()) { + return new File(override.trim()); + } + try { + String configured = com.seleniumboot.internal.SeleniumBootContext.getConfig() + .getFlakiness().getHistoryDir(); + if (configured != null && !configured.isBlank()) { + return new File(configured.trim()); + } + } catch (RuntimeException ignored) { + // no config loaded (or it has no flakiness block): use the default location + } return new File(baseDir(), "metrics-history"); } } diff --git a/src/test/java/com/seleniumboot/unit/ConfigurationLoaderTest.java b/src/test/java/com/seleniumboot/unit/ConfigurationLoaderTest.java index 3ab9e17..60e09d2 100644 --- a/src/test/java/com/seleniumboot/unit/ConfigurationLoaderTest.java +++ b/src/test/java/com/seleniumboot/unit/ConfigurationLoaderTest.java @@ -129,6 +129,17 @@ public void load_noFileButProfileRequested_stillThrows() throws IOException { ConfigurationLoader.load(emptyWorkingDir(), emptyClasspath()); } + @Test + public void load_flakinessHistoryDir_isReadFromTheFile() throws IOException { + Path file = Files.createTempFile("sb-history", ".yml"); + Files.writeString(file, "flakiness:\n historyDir: .sb-history\n"); + System.setProperty("selenium.boot.config", file.toString()); + + SeleniumBootConfig config = ConfigurationLoader.load(); + + assertEquals(config.getFlakiness().getHistoryDir(), ".sb-history"); + } + @Test public void load_emptyFile_usesBuiltInDefaults() throws IOException { Path file = Files.createTempFile("sb-empty", ".yml"); diff --git a/src/test/java/com/seleniumboot/unit/ReportPathsTest.java b/src/test/java/com/seleniumboot/unit/ReportPathsTest.java index 39d0b0f..aea4b2b 100644 --- a/src/test/java/com/seleniumboot/unit/ReportPathsTest.java +++ b/src/test/java/com/seleniumboot/unit/ReportPathsTest.java @@ -18,6 +18,45 @@ public class ReportPathsTest { @AfterMethod public void clearOverride() { System.clearProperty("seleniumboot.reports.dir"); + System.clearProperty("seleniumboot.history.dir"); + } + + @Test + public void historyDir_systemPropertyMovesHistoryOutOfBaseDir() { + System.setProperty("seleniumboot.reports.dir", "target/junit5"); + System.setProperty("seleniumboot.history.dir", "/ci/cache/history"); + assertEquals(ReportPaths.metricsHistoryDir().getPath(), new File("/ci/cache/history").getPath()); + } + + @Test + public void historyDir_configSettingIsUsedWhenNoSystemProperty() { + com.seleniumboot.config.SeleniumBootConfig cfg = new com.seleniumboot.config.SeleniumBootConfig(); + com.seleniumboot.config.SeleniumBootConfig.Flakiness flakiness = + new com.seleniumboot.config.SeleniumBootConfig.Flakiness(); + flakiness.setHistoryDir(" .sb-history "); + cfg.setFlakiness(flakiness); + try (org.mockito.MockedStatic ctx = + org.mockito.Mockito.mockStatic(com.seleniumboot.internal.SeleniumBootContext.class)) { + ctx.when(com.seleniumboot.internal.SeleniumBootContext::getConfig).thenReturn(cfg); + assertEquals(ReportPaths.metricsHistoryDir().getPath(), ".sb-history"); + + System.setProperty("seleniumboot.history.dir", "/from/property"); + assertEquals(ReportPaths.metricsHistoryDir().getPath(), new File("/from/property").getPath(), + "the system property wins over the config"); + } + } + + @Test + public void historyDir_blankConfigKeepsTheDefaultLocation() { + System.clearProperty("seleniumboot.reports.dir"); + com.seleniumboot.config.SeleniumBootConfig cfg = new com.seleniumboot.config.SeleniumBootConfig(); + cfg.setFlakiness(new com.seleniumboot.config.SeleniumBootConfig.Flakiness()); + try (org.mockito.MockedStatic ctx = + org.mockito.Mockito.mockStatic(com.seleniumboot.internal.SeleniumBootContext.class)) { + ctx.when(com.seleniumboot.internal.SeleniumBootContext::getConfig).thenReturn(cfg); + assertEquals(ReportPaths.metricsHistoryDir().getPath(), + "target/metrics-history".replace('/', File.separatorChar)); + } } @Test diff --git a/src/test/java/com/seleniumboot/unit/TestStatusAnalyzerTest.java b/src/test/java/com/seleniumboot/unit/TestStatusAnalyzerTest.java new file mode 100644 index 0000000..de353bc --- /dev/null +++ b/src/test/java/com/seleniumboot/unit/TestStatusAnalyzerTest.java @@ -0,0 +1,164 @@ +package com.seleniumboot.unit; + +import com.seleniumboot.flakiness.TestStatus; +import com.seleniumboot.flakiness.TestStatusAnalyzer; +import com.seleniumboot.flakiness.TestStatusAnalyzer.Outcome; +import org.testng.Assert; +import org.testng.annotations.AfterMethod; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** Retry- and history-aware status: a pure function, then the history-file wiring around it. */ +public class TestStatusAnalyzerTest { + + private static final Outcome FAIL = new Outcome("FAILED", 0); + private static final Outcome PASS = new Outcome("PASSED", 0); + + private static TestStatus classify(Outcome current, String... previous) { + return TestStatusAnalyzer.classify(current, List.of(previous)); + } + + // ------------------------------------------------------------------ pure function + + @Test + public void passedOnRetry_isFlagged_evenWithNoHistory() { + Assert.assertEquals(classify(new Outcome("PASSED", 2)), TestStatus.PASSED_ON_RETRY); + } + + @Test + public void passedFirstTime_getsNoStatus_whateverTheHistory() { + Assert.assertNull(classify(PASS)); + Assert.assertNull(classify(PASS, "FAILED", "PASSED", "FAILED")); + } + + @Test + public void failedAfterEveryEarlierRunPassed_isNewlyFailing() { + Assert.assertEquals(classify(FAIL, "PASSED", "PASSED", "PASSED"), TestStatus.NEWLY_FAILING); + } + + @Test + public void failedAfterASinglePassedRun_isNewlyFailing() { + Assert.assertEquals(classify(FAIL, "PASSED"), TestStatus.NEWLY_FAILING); + } + + @Test + public void failedInEveryEarlierRun_isConsistentlyFailing() { + Assert.assertEquals(classify(FAIL, "FAILED", "FAILED"), TestStatus.CONSISTENTLY_FAILING); + } + + @Test + public void failedTwiceInARow_isNotYetConsistent() { + Assert.assertNull(classify(FAIL, "FAILED")); + } + + @Test + public void failedWithMixedHistory_isIntermittent() { + Assert.assertEquals(classify(FAIL, "PASSED", "FAILED", "PASSED"), TestStatus.INTERMITTENT); + } + + @Test + public void failedWithNoHistory_hasNoStatus() { + Assert.assertNull(classify(FAIL)); + Assert.assertNull(TestStatusAnalyzer.classify(FAIL, null)); + } + + @Test + public void failedAfterRetries_isStillJudgedByHistory() { + Assert.assertEquals(classify(new Outcome("FAILED", 2), "PASSED", "PASSED"), TestStatus.NEWLY_FAILING); + } + + @Test + public void skippedRunsInHistory_areNotEvidence() { + Assert.assertEquals(classify(FAIL, "SKIPPED", "PASSED", "SKIPPED"), TestStatus.NEWLY_FAILING); + Assert.assertNull(classify(FAIL, "SKIPPED", "SKIPPED")); + } + + @Test + public void skippedOrMissingCurrent_hasNoStatus() { + Assert.assertNull(classify(new Outcome("SKIPPED", 0), "PASSED")); + Assert.assertNull(TestStatusAnalyzer.classify(null, List.of("PASSED"))); + } + + // ------------------------------------------------------------------ history files + + private Path dir; + private String previousProperty; + + @BeforeMethod + public void setUp() throws Exception { + dir = Files.createTempDirectory("status-history"); + previousProperty = System.getProperty("seleniumboot.reports.dir"); + System.setProperty("seleniumboot.reports.dir", dir.toString()); + Files.createDirectories(dir.resolve("metrics-history")); + } + + @AfterMethod + public void tearDown() throws Exception { + if (previousProperty == null) System.clearProperty("seleniumboot.reports.dir"); + else System.setProperty("seleniumboot.reports.dir", previousProperty); + File[] files = dir.resolve("metrics-history").toFile().listFiles(); + if (files != null) for (File f : files) f.delete(); + Files.deleteIfExists(dir.resolve("metrics-history")); + Files.deleteIfExists(dir); + } + + private void run(String name, long modified, String... idStatus) throws Exception { + StringBuilder sb = new StringBuilder("{\"tests\":["); + for (int i = 0; i < idStatus.length; i += 2) { + if (i > 0) sb.append(','); + sb.append("{\"testId\":\"").append(idStatus[i]).append("\",\"status\":\"") + .append(idStatus[i + 1]).append("\"}"); + } + Path f = dir.resolve("metrics-history").resolve(name); + Files.writeString(f, sb.append("]}").toString()); + Assert.assertTrue(f.toFile().setLastModified(modified)); + } + + @Test + public void emptyHistoryDirectory_onlyRetriesAreFlagged() { + Map r = TestStatusAnalyzer.forRun( + Map.of("A.a", FAIL, "B.b", new Outcome("PASSED", 1)), Set.of()); + Assert.assertEquals(r, Map.of("B.b", TestStatus.PASSED_ON_RETRY)); + } + + @Test + public void historyIsReadPerTest() throws Exception { + run("r1.json", 1000, "A.a", "PASSED", "B.b", "FAILED", "C.c", "PASSED"); + run("r2.json", 2000, "A.a", "PASSED", "B.b", "FAILED", "C.c", "FAILED"); + run("r3.json", 3000, "A.a", "PASSED", "B.b", "FAILED", "C.c", "PASSED"); + + Map r = TestStatusAnalyzer.forRun( + Map.of("A.a", FAIL, "B.b", FAIL, "C.c", FAIL, "D.d", FAIL), Set.of()); + + Assert.assertEquals(r.get("A.a"), TestStatus.NEWLY_FAILING); + Assert.assertEquals(r.get("B.b"), TestStatus.CONSISTENTLY_FAILING); + Assert.assertEquals(r.get("C.c"), TestStatus.INTERMITTENT); + Assert.assertNull(r.get("D.d"), "a test with no history gets no status"); + } + + @Test + public void historyWrittenByThisRun_isNotComparedWithItself() throws Exception { + run("earlier.json", 1000, "A.a", "PASSED"); + run("this-run.json", 2000, "A.a", "FAILED"); + + Assert.assertEquals(TestStatusAnalyzer.forRun(Map.of("A.a", FAIL), Set.of()).get("A.a"), + TestStatus.INTERMITTENT, "sanity: without the exclusion this run is counted"); + Assert.assertEquals(TestStatusAnalyzer.forRun(Map.of("A.a", FAIL), Set.of("this-run.json")).get("A.a"), + TestStatus.NEWLY_FAILING); + } + + @Test + public void unreadableHistoryFile_isSkipped() throws Exception { + Files.writeString(dir.resolve("metrics-history").resolve("broken.json"), "{ not json"); + run("ok.json", 5000, "A.a", "PASSED"); + Assert.assertEquals(TestStatusAnalyzer.forRun(Map.of("A.a", FAIL), Set.of()).get("A.a"), + TestStatus.NEWLY_FAILING); + } +} diff --git a/src/test/java/com/seleniumboot/unit/TestStatusExportTest.java b/src/test/java/com/seleniumboot/unit/TestStatusExportTest.java new file mode 100644 index 0000000..70e758d --- /dev/null +++ b/src/test/java/com/seleniumboot/unit/TestStatusExportTest.java @@ -0,0 +1,85 @@ +package com.seleniumboot.unit; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.seleniumboot.metrics.ExecutionMetrics; +import org.testng.Assert; +import org.testng.annotations.AfterMethod; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; + +/** exportToJson() writes {@code testStatus}, comparing against history without counting itself. */ +public class TestStatusExportTest { + + private Path dir; + private String previousProperty; + + @BeforeMethod + public void setUp() throws Exception { + ExecutionMetrics.reset(); + dir = Files.createTempDirectory("status-export"); + previousProperty = System.getProperty("seleniumboot.reports.dir"); + System.setProperty("seleniumboot.reports.dir", dir.toString()); + Path history = Files.createDirectories(dir.resolve("metrics-history")); + Path earlier = history.resolve("earlier.json"); + Files.writeString(earlier, "{\"tests\":[{\"testId\":\"p.A.fresh\",\"status\":\"PASSED\"}," + + "{\"testId\":\"p.A.steady\",\"status\":\"PASSED\"}]}"); + Assert.assertTrue(earlier.toFile().setLastModified(1000)); + } + + @AfterMethod + public void tearDown() throws Exception { + ExecutionMetrics.reset(); + if (previousProperty == null) System.clearProperty("seleniumboot.reports.dir"); + else System.setProperty("seleniumboot.reports.dir", previousProperty); + File[] history = dir.resolve("metrics-history").toFile().listFiles(); + if (history != null) for (File f : history) f.delete(); + Files.deleteIfExists(dir.resolve("metrics-history")); + Files.deleteIfExists(dir.resolve("selenium-boot-metrics.json")); + Files.deleteIfExists(dir); + } + + private static void record(String id, String status, int retries) { + ExecutionMetrics.markStart(id); + for (int i = 0; i < retries; i++) ExecutionMetrics.recordRetry(id); + ExecutionMetrics.recordStatus(id, status); + ExecutionMetrics.markEnd(id); + } + + private JsonNode export() throws Exception { + ExecutionMetrics.exportToJson(); + return new ObjectMapper().readTree(dir.resolve("selenium-boot-metrics.json").toFile()).path("tests"); + } + + private static String statusOf(JsonNode tests, String id) { + for (JsonNode t : tests) { + if (id.equals(t.path("testId").asText())) return t.path("testStatus").asText(null); + } + throw new AssertionError("no test " + id); + } + + @Test + public void exportedMetrics_carryStatusFromRetriesAndHistory() throws Exception { + record("p.A.fresh", "FAILED", 0); + record("p.A.steady", "PASSED", 0); + record("p.A.retried", "PASSED", 1); + + JsonNode tests = export(); + + Assert.assertEquals(statusOf(tests, "p.A.fresh"), "NEWLY_FAILING"); + Assert.assertNull(statusOf(tests, "p.A.steady")); + Assert.assertEquals(statusOf(tests, "p.A.retried"), "PASSED_ON_RETRY"); + } + + @Test + public void exportingTwiceInOneRun_doesNotCompareTheRunWithItself() throws Exception { + record("p.A.fresh", "FAILED", 0); + export(); + // The first export left a copy of this run in the history directory. + Assert.assertEquals(statusOf(export(), "p.A.fresh"), "NEWLY_FAILING"); + } +} diff --git a/src/test/java/com/seleniumboot/unit/TestStatusReportTest.java b/src/test/java/com/seleniumboot/unit/TestStatusReportTest.java new file mode 100644 index 0000000..431af8e --- /dev/null +++ b/src/test/java/com/seleniumboot/unit/TestStatusReportTest.java @@ -0,0 +1,65 @@ +package com.seleniumboot.unit; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.seleniumboot.reporting.HtmlReportGenerator; +import org.testng.Assert; +import org.testng.annotations.Test; + +import java.lang.reflect.Method; + +/** Retry/history status in the report: a section by the Flakiness Radar, newly failing first. */ +public class TestStatusReportTest { + + private static String call(String method, String json, Object... extra) throws Exception { + JsonNode tests = new ObjectMapper().readTree(json); + Class[] types = extra.length == 0 + ? new Class[]{JsonNode.class} + : new Class[]{JsonNode.class, boolean.class}; + Method m = HtmlReportGenerator.class.getDeclaredMethod(method, types); + m.setAccessible(true); + Object[] args = extra.length == 0 ? new Object[]{tests} : new Object[]{tests, extra[0]}; + return (String) m.invoke(null, args); + } + + private static String test(String id, String cls, String status, String testStatus) { + return "{\"testId\":\"" + id + "\",\"testClassName\":\"" + cls + "\",\"status\":\"" + status + "\"" + + (testStatus == null ? "" : ",\"testStatus\":\"" + testStatus + "\"") + "}"; + } + + @Test + public void section_listsFlaggedTestsWithNewlyFailingFirst() throws Exception { + String html = call("buildTestStatusSection", "[" + + test("p.A.retry", "A", "PASSED", "PASSED_ON_RETRY") + "," + + test("p.B.fresh", "B", "FAILED", "NEWLY_FAILING") + "," + + test("p.C.plain", "C", "PASSED", null) + "]"); + Assert.assertTrue(html.contains("Newly failing") && html.contains("Passed on retry"), html); + Assert.assertTrue(html.contains("fresh") && html.contains("retry"), html); + Assert.assertFalse(html.contains("plain"), html); + Assert.assertTrue(html.indexOf("Newly failing") < html.indexOf("Passed on retry"), html); + } + + @Test + public void section_isEmptyWhenNothingIsFlagged() throws Exception { + Assert.assertEquals(call("buildTestStatusSection", "[" + test("p.A.a", "A", "PASSED", null) + "]"), ""); + } + + @Test + public void failureList_putsNewlyFailingFirst() throws Exception { + String html = call("buildFailureRows", "[" + + test("p.A.old", "ClassA", "FAILED", "CONSISTENTLY_FAILING") + "," + + test("p.B.fresh", "ClassB", "FAILED", "NEWLY_FAILING") + "," + + test("p.A.plain", "ClassA", "FAILED", null) + "]", false); + int fresh = html.indexOf("ClassB"); + int a = html.indexOf("ClassA"); + Assert.assertTrue(fresh >= 0 && a >= 0 && fresh < a, "ClassB (newly failing) must come first: " + html); + } + + @Test + public void failureList_keepsOrderWhenNothingIsNewlyFailing() throws Exception { + String html = call("buildFailureRows", "[" + + test("p.A.a", "ClassA", "FAILED", null) + "," + + test("p.B.b", "ClassB", "FAILED", "CONSISTENTLY_FAILING") + "]", false); + Assert.assertTrue(html.indexOf("ClassA") < html.indexOf("ClassB"), html); + } +}