Skip to content

Report the retry budget in assertion failure messages #5223

Description

@aaltat

Is your feature request related to a problem? Please describe.

When an assertion keyword fails, the failure message says nothing about the fact
that the assertion was retried, or for how long. Get Element States id=not_here contains hidden
fails with:

Elements states '['detached']' (list) should contain '['hidden']' (list)

That message is identical whether the assertion failed instantly or was retried for a
full second before giving up. There is no hint that retrying happened, that it is
configurable, or that Set Retry Assertions For exists.

This is a discoverability problem more than a debugging one. Users who do not already
know about retry_assertions_for have no path to discovering it, because the one moment
they would care about it — an assertion that failed after quietly retrying — is exactly
the moment the library stays silent. Anecdotally, experienced Robot Framework users are
surprised the setting exists at all.

The information is already computed. Browser/assertion_engine.py logs it at DEBUG in the
finally block of with_assertion_polling:

Assertion polling statistics:
First element asserted in: 0.2534 seconds
Total tries: 4
Elapsed time in retries 0.7864 seconds

DEBUG is the wrong level for something a user needs in order to know a feature exists.

Describe the solution you'd like

Include the retry budget in the assertion failure message itself, at the two raise sites
in with_assertion_polling (Browser/assertion_engine.py).

Gate it on retries having actually happened (tries > 1). That is both the only case where
the information is useful and the thing that keeps the change small: an assertion that failed
on its first attempt, or one running with retry_assertions_for=0, keeps its message
byte-for-byte.

Describe alternatives you've considered

  • Leave it at DEBUG. Status quo. Does not solve discoverability — nobody turns on DEBUG
    before they know there is something to look for.
  • Document it harder. Does not reach the user at the moment they need it.
  • Always append, ungated. Larger compatibility surface for no extra benefit, since a
    non-retried assertion has no retry budget worth reporting.

Additional context

Compatibility. Appending to assertion messages is a breaking change for
Run Keyword And Expect Error patterns that match exactly. Measured in this repository's
own acceptance tests:

Run Keyword And Expect Error sites 225
pattern uses a glob / regexp 141
exact match 84

The tries > 1 gate narrows the affected set considerably — the exact-match sites are
mostly non-retrying errors (Error: Crash, plugin errors) rather than assertion-keyword
failures. The residual should be measured before implementing. The same consideration
applies to users' suites, which is why this needs release notes.

Design question — where the information goes. The blast radius depends on placement, and
that is worth deciding deliberately rather than defaulting to "append to the end":

  • appended as a trailing sentence — simplest, but touches the message that exact matches see;
  • a separate WARN/INFO log line beside the failure — zero message compatibility cost,
    but easier for a user to miss;
  • attached only when the assertion exhausted its budget (as opposed to any retry at all).

Secondary benefit. retry_assertions_for is currently the one scope setting whose effect
cannot be verified except with a stopwatch. atest/test/08_Scope_Tests measures wall-clock
duration to infer it, and that is the source of roughly 20 CI failures across the last ~1200
runs. Every other scope setting can be verified from a message — Playwright names the timeout
it used (Timeout 1500ms exceeded), and strict mode raises strict mode violation. Naming
the retry budget in the failure would let those tests assert on a message too, and drop the
last timing-based assertion in that suite.

Related: Browser/assertion_engine.py, atest/test/08_Scope_Tests/scope_keywords.resource.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acknowledgeTo be acknowledged in release notesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions