Is your feature request related to a problem? Please describe.
When an assertion keyword fails, the failure message says nothing about the fact
that the assertion was retried, or for how long. Get Element States id=not_here contains hidden
fails with:
Elements states '['detached']' (list) should contain '['hidden']' (list)
That message is identical whether the assertion failed instantly or was retried for a
full second before giving up. There is no hint that retrying happened, that it is
configurable, or that Set Retry Assertions For exists.
This is a discoverability problem more than a debugging one. Users who do not already
know about retry_assertions_for have no path to discovering it, because the one moment
they would care about it — an assertion that failed after quietly retrying — is exactly
the moment the library stays silent. Anecdotally, experienced Robot Framework users are
surprised the setting exists at all.
The information is already computed. Browser/assertion_engine.py logs it at DEBUG in the
finally block of with_assertion_polling:
Assertion polling statistics:
First element asserted in: 0.2534 seconds
Total tries: 4
Elapsed time in retries 0.7864 seconds
DEBUG is the wrong level for something a user needs in order to know a feature exists.
Describe the solution you'd like
Include the retry budget in the assertion failure message itself, at the two raise sites
in with_assertion_polling (Browser/assertion_engine.py).
Gate it on retries having actually happened (tries > 1). That is both the only case where
the information is useful and the thing that keeps the change small: an assertion that failed
on its first attempt, or one running with retry_assertions_for=0, keeps its message
byte-for-byte.
Describe alternatives you've considered
- Leave it at DEBUG. Status quo. Does not solve discoverability — nobody turns on DEBUG
before they know there is something to look for.
- Document it harder. Does not reach the user at the moment they need it.
- Always append, ungated. Larger compatibility surface for no extra benefit, since a
non-retried assertion has no retry budget worth reporting.
Additional context
Compatibility. Appending to assertion messages is a breaking change for
Run Keyword And Expect Error patterns that match exactly. Measured in this repository's
own acceptance tests:
Run Keyword And Expect Error sites |
225 |
| pattern uses a glob / regexp |
141 |
| exact match |
84 |
The tries > 1 gate narrows the affected set considerably — the exact-match sites are
mostly non-retrying errors (Error: Crash, plugin errors) rather than assertion-keyword
failures. The residual should be measured before implementing. The same consideration
applies to users' suites, which is why this needs release notes.
Design question — where the information goes. The blast radius depends on placement, and
that is worth deciding deliberately rather than defaulting to "append to the end":
- appended as a trailing sentence — simplest, but touches the message that exact matches see;
- a separate
WARN/INFO log line beside the failure — zero message compatibility cost,
but easier for a user to miss;
- attached only when the assertion exhausted its budget (as opposed to any retry at all).
Secondary benefit. retry_assertions_for is currently the one scope setting whose effect
cannot be verified except with a stopwatch. atest/test/08_Scope_Tests measures wall-clock
duration to infer it, and that is the source of roughly 20 CI failures across the last ~1200
runs. Every other scope setting can be verified from a message — Playwright names the timeout
it used (Timeout 1500ms exceeded), and strict mode raises strict mode violation. Naming
the retry budget in the failure would let those tests assert on a message too, and drop the
last timing-based assertion in that suite.
Related: Browser/assertion_engine.py, atest/test/08_Scope_Tests/scope_keywords.resource.
Is your feature request related to a problem? Please describe.
When an assertion keyword fails, the failure message says nothing about the fact
that the assertion was retried, or for how long.
Get Element States id=not_here contains hiddenfails with:
That message is identical whether the assertion failed instantly or was retried for a
full second before giving up. There is no hint that retrying happened, that it is
configurable, or that
Set Retry Assertions Forexists.This is a discoverability problem more than a debugging one. Users who do not already
know about
retry_assertions_forhave no path to discovering it, because the one momentthey would care about it — an assertion that failed after quietly retrying — is exactly
the moment the library stays silent. Anecdotally, experienced Robot Framework users are
surprised the setting exists at all.
The information is already computed.
Browser/assertion_engine.pylogs it at DEBUG in thefinallyblock ofwith_assertion_polling:DEBUG is the wrong level for something a user needs in order to know a feature exists.
Describe the solution you'd like
Include the retry budget in the assertion failure message itself, at the two
raisesitesin
with_assertion_polling(Browser/assertion_engine.py).Gate it on retries having actually happened (
tries > 1). That is both the only case wherethe information is useful and the thing that keeps the change small: an assertion that failed
on its first attempt, or one running with
retry_assertions_for=0, keeps its messagebyte-for-byte.
Describe alternatives you've considered
before they know there is something to look for.
non-retried assertion has no retry budget worth reporting.
Additional context
Compatibility. Appending to assertion messages is a breaking change for
Run Keyword And Expect Errorpatterns that match exactly. Measured in this repository'sown acceptance tests:
Run Keyword And Expect ErrorsitesThe
tries > 1gate narrows the affected set considerably — the exact-match sites aremostly non-retrying errors (
Error: Crash, plugin errors) rather than assertion-keywordfailures. The residual should be measured before implementing. The same consideration
applies to users' suites, which is why this needs release notes.
Design question — where the information goes. The blast radius depends on placement, and
that is worth deciding deliberately rather than defaulting to "append to the end":
WARN/INFOlog line beside the failure — zero message compatibility cost,but easier for a user to miss;
Secondary benefit.
retry_assertions_foris currently the one scope setting whose effectcannot be verified except with a stopwatch.
atest/test/08_Scope_Testsmeasures wall-clockduration to infer it, and that is the source of roughly 20 CI failures across the last ~1200
runs. Every other scope setting can be verified from a message — Playwright names the timeout
it used (
Timeout 1500ms exceeded), and strict mode raisesstrict mode violation. Namingthe retry budget in the failure would let those tests assert on a message too, and drop the
last timing-based assertion in that suite.
Related:
Browser/assertion_engine.py,atest/test/08_Scope_Tests/scope_keywords.resource.