Skip to content

Auto-fix: spec failure in yaml-examples/triage-fixtures/01-spec-drift.test.yaml - #17

Open
feng-shiplight wants to merge 1 commit into
mainfrom
triage-autofix/yaml-examples-triage-fixtures-01-spec-drift-test-yaml-f1235ca53966
Open

Auto-fix: spec failure in yaml-examples/triage-fixtures/01-spec-drift.test.yaml#17
feng-shiplight wants to merge 1 commit into
mainfrom
triage-autofix/yaml-examples-triage-fixtures-01-spec-drift-test-yaml-f1235ca53966

Conversation

@feng-shiplight

Copy link
Copy Markdown
Contributor

Fix stale locator in 01-spec-drift triage fixture

What failed

The test triage-fixtures/01-spec-drift.test.yaml verifies that the report
action button is visible on the dashboard. Its VERIFY step used a js:
predicate pinned to the old element id:

- VERIFY: The "Generate Report" button is visible on the dashboard
  js: "await expect(page.locator('#generateReport')).toBeVisible({ timeout: 3000 })"

Because #generateReport no longer exists in the DOM, the visibility
assertion times out after 3000 ms and the test fails.

Root cause

The dashboard was renamed in a UI refresh — the functionality is unchanged,
only the label and id changed:

Old label Old id New label New id
Generate Report generateReport Create Report createReport

The application behavior is correct: the report action button is present
and visible; it is simply labeled "Create Report" now. Confirmed by inspecting
the live page
https://static.shiplight.ai/testing/failure-analysis/renamed-elements.html,
whose rendered DOM shows a <button> reading "Create Report". This is a
stale-spec (assertion-drift) failure, so the fix is test-only.

The change

Updated only the VERIFY step to target the renamed button:

- VERIFY: The "Create Report" button is visible on the dashboard
  js: "await expect(page.locator('#createReport')).toBeVisible({ timeout: 3000 })"
  • Minimal: a single statement changed — the id locator and the matching
    statement text. No other steps, templates, or files touched.
  • Robust: retains the deterministic toBeVisible({ timeout: 3000 })
    Playwright web-assertion, which polls until the element is visible rather
    than relying on a fixed sleep. The locator now matches the app's current,
    stable id (#createReport).

Verification (recorded, not a merge gate):Verified — re-ran yaml-examples/triage-fixtures/01-spec-drift.test.yaml twice, both passed.

Diff
diff --git a/yaml-examples/triage-fixtures/01-spec-drift.test.yaml b/yaml-examples/triage-fixtures/01-spec-drift.test.yaml
index 5a46d41..29ed63c 100644
--- a/yaml-examples/triage-fixtures/01-spec-drift.test.yaml
+++ b/yaml-examples/triage-fixtures/01-spec-drift.test.yaml
@@ -27,5 +27,5 @@ statements:
   - URL: /testing/failure-analysis/renamed-elements.html
 
   # Stale: the button carries id `createReport` and reads "Create Report" now.
-  - VERIFY: The "Generate Report" button is visible on the dashboard
-    js: "await expect(page.locator('#generateReport')).toBeVisible({ timeout: 3000 })"
+  - VERIFY: The "Create Report" button is visible on the dashboard
+    js: "await expect(page.locator('#createReport')).toBeVisible({ timeout: 3000 })"

Failing run: https://github.com/ShiplightAI/examples/actions/runs/31583416180

Auto-generated by CI Failure Triage. Human review required — do not auto-merge. Decide based on the verification status above.

…yaml (verification: verified)

Update the VERIFY step to target the renamed button: change the js predicate locator from '#generateReport' to '#createReport' and update the statement text from 'Generate Report' to 'Create Report'.

Opened automatically by CI Failure Triage; verification result recorded in the PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant