Skip to content

feat(failure-intel): add bounded failure detail records - #153

Merged
shossain786 merged 3 commits into
seleniumboot:masterfrom
adity982:feat/failure-detail-record
Oct 14, 2026
Merged

shossain786 merged 3 commits into
seleniumboot:masterfrom
adity982:feat/failure-detail-record

Conversation

@adity982

Copy link
Copy Markdown
Contributor

Summary

  • add one bounded failureDetail record for each failed test and export it to metrics JSON
  • render that same record in the HTML failure panel and reuse it in the opt-in AI prompt
  • capture locator, application frame, last successful step, page context, heal events, console errors, test status, and same-cause representative context only when available
  • document fixed limits and the exact data sent for AI analysis

Safety and compatibility

  • missing fields are omitted rather than invented
  • text/list/URL inputs are bounded before metrics or prompt use
  • captured diagnostic values are explicitly treated as untrusted evidence in the AI prompt
  • raw errors and stack traces remain available
  • AI remains opt-in, timeout-bounded, and unable to change the suite result
  • no new dependency or stable API signature change

Validation

  • mvn test — 669 tests passed
  • focused failure-detail, export, report, prompt, metrics, and grouping regressions passed
  • mvn -DskipTests compile passed
  • git diff --check passed

Closes #142

@shossain786

Copy link
Copy Markdown
Contributor

Thanks, Aditya, this is a well-scoped slice. I read the whole diff and ran the suite on the branch: 669 run, 0 failures, matching your numbers. (CI hasn't run on this PR yet because it's from a first-time fork, so the local run is the evidence.)

What I checked:

  • One FailureDetail record feeds the metrics JSON, the HTML panel and the AI prompt. Fields are omitted rather than invented, and the HTML path escapes captured values (there's a test for it).
  • The AI path is still opt-in and timeout-bounded, and the docs list exactly what is sent, including console output and URLs.
  • Moving the screenshot capture ahead of the AI call and the hooks is fine, and needed so the group representative has a screenshot.

One thing I'd like changed before merge:

  1. failingLocator can name the wrong locator. Locator.whenReady and WaitEngine record the locator on any wait timeout, including ones the test catches and recovers from (an optional banner, an isVisible-style probe). FailureDetail.from then prefers that recorded value over the one parsed from the error message. A test that swallows a missing optional element and later fails on an assertion will report the optional element as the failing locator, which is worse than reporting none. Either prefer the locator parsed from the final error message and use the recorded one only as a fallback when the final failure is a locator/wait timeout, or record only when the exception actually escapes. Please add a test for the "caught timeout, then unrelated failure" case.

Smaller, not blockers:

Happy to merge once 1 is in.

@adity982

Copy link
Copy Markdown
Contributor Author

Addressed the failing-locator issue in exact head 7c14eac after merging current master (including #150):

  • a locator parsed from the final error now takes precedence;
  • a previously recorded locator is used only for a final locator/wait failure from Locator or WaitEngine;
  • a caught optional-element timeout followed by an unrelated assertion now omits failingLocator instead of reporting the stale optional locator;
  • the export fixture now represents a real final locator failure.

Validation: 6 focused failure-detail/export tests passed, then the full Maven suite passed 693/693; git diff --check passed. Marking the PR ready as requested.

@adity982
adity982 marked this pull request as ready for review October 14, 2026 03:07
@shossain786
shossain786 merged commit 62022f3 into seleniumboot:master Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(failure-intel): detail record per failure, reused by AI analysis

2 participants