Skip to content

feat(wda): an element check takes its fields from the find's answer - #263

Open
luka-vladika-happening-xyz wants to merge 1 commit into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:feat/wda-element-fields-from-find
Open

luka-vladika-happening-xyz wants to merge 1 commit into
devicelab-dev:mainfrom
luka-vladika-happening-xyz:feat/wda-element-fields-from-find

Conversation

@luka-vladika-happening-xyz

@luka-vladika-happening-xyz luka-vladika-happening-xyz commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #258. An element check on the WDA driver sent the find and then four more requests (GET /element/{id}/text, /rect, /displayed and /name). WebDriverAgent answers them one after another, and each resolves the element again. The session now asks WebDriverAgent to put those four fields in every find's answer, and the checks take them from there. On a real iPhone, a check by id on a busy page went from 7.4 s to 2.45 s.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Changes Made

  • CreateSession sends shouldUseCompactResponses: false and elementResponseAttributes: "name,text,rect,displayed" in the same /appium/settings call as snapshotMaxDepth. WebDriverAgent ignores a setting it does not know.
  • Client.FindElementWithFields and FindElementsWithFields return each element's id with its fields (Element, ElementFields), or with none when the answer lacks any of the four. FindElement and FindElements keep their signatures for the other callers.
  • The id is read from ELEMENT, else from the first string value that is not a field. The W3C fallback took the first string value it met, which could now be the element's name or text.
  • getElementInfo(id) now calls elementInfo(Element). With the fields it sends no request. Without them it reads them as before. The visibility rule after that is unchanged.
  • The finds that feed it pass the fields: findElementByWDA (a regex id, the text predicate), findByIDQuery and onScreenOf, findInteractiveElementByWDA, and the exact-text predicate in findElementForTapWithContext. Its contains-predicate fallback (step 4, after a page source) still reads the element again, since the page source came in between.
  • Tests (element_fields_test.go with a fake WDA, and TestCreateSession):
    • TestFindAnswerFieldsSkipElementReads: by text and by id, no element GET when the answer has the fields, and the same type, text, bounds and visibility as the GETs would give;
    • TestFindAnswerWithoutFieldsReadsThem: an answer without them, so the four GETs run;
    • TestElementFromAnswer: the id from the W3C key and not from a field, a null text read as "", and no fields when the rect lacks a number;
    • TestCreateSession checks that the two settings are sent.

Related Issues

Closes #258

Testing

  • The new tests fail on main: element_fields_test.go does not compile there, and TestCreateSession fails on the missing settings
  • go vet, gofmt -s -l and golangci-lint v2.13.2 (CI's pin, under go1.26.7) report nothing
  • go test -race ./... passes. pkg/driver/browser/cdp and pkg/emulator were left out: they fail on this Mac on main too (Chrome's launch; an emulator holding 5554)
  • On a real iPhone 17 (iOS 27.0.1, WebDriverAgent 11.4.1 through --wda-url), the same build with and without this change, alternated over four rounds, medians:
    • assertVisible by id on an app's busy page (about 2,900 elements): 7.4 s → 2.45 s. The find took 1.9–3.0 s either way; the four reads were the rest;
    • by id on a lighter screen of the same app: 1.24 s → 0.42 s; on the Settings app: 0.58 s → 0.18 s;
    • a whole flow on that app (it opens an event from the home screen and taps an odd on its page): 170 s → 151 s;
    • no run with the change sent an element GET, and all of its runs passed.
  • make test as a whole: not run, because its device tests drive whatever device is attached

Checklist

  • Code follows project style guidelines
  • Self-reviewed the code
  • Added/updated documentation as needed (no CHANGELOG line: you add it when merging)
  • No breaking changes (or documented if breaking)
  • CHANGELOG.md updated (left to you when merging)

Additional Notes

  • The fields are the GETs' values. name is wdType, as GET /name answers. text is the value, else the label, as GET /text; it is null when both are empty, which is read as "", what GET /text answers then. rect is wdRect, and displayed is wdVisible, as GET /displayed. They are read from the find's own snapshot (FBDictionaryResponseWithElement in FBResponsePayload.m). The code is the same in WebDriverAgent 11.4.1 and in the 16.17.5 this repository bundles.
  • WebDriverAgent keeps these settings process-wide (FBConfiguration), not per session. A later client of the same WebDriverAgent (--reuse-wda, --wda-url) gets the longer answers too. They still carry ELEMENT and the W3C key, so a client that reads the id is not affected.
  • Many matches: a find now computes the fields of every match it returns. With one match it took the same time with and without them. onScreenOf read at most maxIDCandidates (5) elements, while a find with dozens of same-id matches now computes the fields for all of them. Measured since (comment): the fields add nothing measurable to the find, even at 164 matches.

getElementInfo sent four requests after each find (GET /element/{id}/text,
/rect, /displayed and /name). WebDriverAgent answers them one at a time,
and each resolves the element again, so on a real iPhone a check cost the
find plus four round trips; on a long list a one-match check took about
2 s.

The session now asks WebDriverAgent to put those four fields in every
find's answer (shouldUseCompactResponses: false, elementResponseAttributes
"name,text,rect,displayed", sent with snapshotMaxDepth). They are the same
values the GETs return, read from the find's own snapshot
(FBDictionaryResponseWithElement). The finds that feed getElementInfo take
them from the answer and send no further request; when an answer lacks any
of them (a WebDriverAgent that ignored the settings), the four GETs run as
before. The id is read from ELEMENT or the W3C key and never from a field.

Refs devicelab-dev#258
@codecov

codecov Bot commented Nov 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.26168% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/driver/wda/driver.go 92.45% 3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@luka-vladika-happening-xyz

Copy link
Copy Markdown
Contributor Author

I measured the many-matches case the description left open. Same iPhone 17 (iOS 27.0.1, WebDriverAgent 11.4.1), the Settings app's main screen, raw WDA calls, medians of three rounds in alternating order. "Before" is the compact find followed by this driver's reads (at most maxIDCandidates matches, four GETs each):

Class chain Matches Before With the fields
label == 'General' 2 1.15 s 0.29 s
XCUIElementTypeCell 11 3.21 s 1.34 s
XCUIElementTypeStaticText 18 3.89 s 1.31 s
XCUIElementTypeAny 164 19.69 s 15.31 s

The fields add nothing measurable to the find, even at 164 matches: the compact find alone took 16.4 s, and the find with the fields 15.3 s. WebDriverAgent takes each match's snapshot for its id anyway (FBDictionaryResponseWithElement), and the fields come from that same snapshot.

@omnarayan

Copy link
Copy Markdown
Contributor

Thanks for this. Testing it for 1.1.31, I hit a regression on the simulator that I can't explain from the runner side, so I'm holding this PR until we understand it.

What happens

TestHive iOS login on WDA (iPhone 17 Pro simulator, iOS 27.0): tap the username field, type, then tapOn: "Password". When the keyboard comes up, the app moves the form up by about 147 pt.

  • Without this PR: GET /element/{id}/rect for the password field returns y≈377 (centre 399.5), and the tap focuses the field. 4/4 on both booted and cold runs.
  • With this PR: the find's answer puts the field at y≈524 (centre 546.5), its pre-keyboard position. The tap lands on "Forgot Password?" and opens its alert, and the next inputText has nothing focused. WDA runs on the booted simulator: 3/4, then 2/4. Cold runs: 3/4, then 4/4.

What I tried

First I thought only the rect in the find's answer was old, so I had the tap re-read the rect with GET /rect before tapping. That didn't help: GET /rect returned the same old y≈524 for an element found by a find started 2–4 s after the keyboard was up. The runner's GET /rect code is unchanged by this PR. My guess is that the session settings this PR turns on (shouldUseCompactResponses: false + elementResponseAttributes, which WDA keeps for the whole process) make WDA answer with the snapshot taken at find time, even for the later /rect. I haven't confirmed that in WDA's source.

On RNTester (48 flows, WDA) the results were the same with and without the PR, so I only see this on a screen whose layout moves after the find.

Could you check on your devices whether /rect after a find with these settings follows a layout change (keyboard pushing a form up is the easy case)? If it doesn't, a tap probably needs a position that doesn't come from these settings.

@luka-vladika-happening-xyz

luka-vladika-happening-xyz commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for testing it. I checked your question on a real iPhone 17 (iOS 27.0.1, WebDriverAgent 11.4.1), with raw WDA calls on the Settings app, three rounds of each mode in alternating order: a session, the mode's settings, a find of the "General" row by accessibility id, GET /rect of it, a layout change, then GET /rect of the same element id 1-2.5 s later and a new find.

Layout change shouldUseCompactResponses: true (before this PR) false + elementResponseAttributes (this PR)
the list dragged up (the row moves from y 804 to 498) /rect after: 498, 3 of 3 498, 3 of 3 (the find's answer said 804, its position at find time)
the search field tapped, the keyboard up (the row moves from 804 to 636) /rect after: 636, 3 of 3 636, 3 of 3

So on this device /rect after a find follows the layout with the PR's settings exactly as without them, and after the drag a new find answers the new position (498) in both modes. That matches WebDriverAgent's code as I read it: handleGetRect reads wdRect through fb_standardSnapshot, a new snapshot each time, and FBResponseWithCachedElement clears lastSnapshot after building the answer, so the settings only add fields to the find's answer.

I could not reproduce the old position, so I can't explain what your simulator did. The one thing the PR changes in time: the tap now uses the rect from the find's own snapshot, where before it was read by four more requests, roughly 100-300 ms later on a simulator. If the form was still moving when the find ran, the PR's rect is where it was then, and the old /rect was where it had got to. That doesn't fit your re-read /rect returning the old y too, unless that re-read came right after the find. If you can share the flow (or the screen's layout), I'll run it on our phones and simulators with and without the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] WDA driver: an element's text, rect and visibility from the find's answer instead of four more requests

3 participants