Skip to content

fix(wda): elements inside a WebView keep their screen coordinates - #139

Open
cosinec wants to merge 1 commit into
Derrors:mainfrom
cosinec:fix/webview-widget-offset
Open

cosinec wants to merge 1 commit into
Derrors:mainfrom
cosinec:fix/webview-widget-offset

Conversation

@cosinec

@cosinec cosinec commented Nov 16, 2026

Copy link
Copy Markdown

What and why

parse_tree treats a node at (0,0) that has its parent's size, under a parent not at (0,0), as the start of a widget's relative coordinates, and adds the parent's position to everything below it. Inside a WKWebView there is a node exactly like that: it reports (0,0) at the WebView's own size. The web elements under it, though, report screen coordinates. So when the WebView sits below a navigation bar (an app's in-app browser, WebView at y=100), every element inside came out 100 pt too low, and iphone_tap_element tapped the row underneath. Safari's WebView starts at y=0, which is why it never showed there.

A WebView and everything inside it are now left out of the widget heuristic (Open.in_web). Widgets outside web content are unchanged.

Tests

1. Unit tests

  • cargo test --workspace --all-features --locked, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings and cargo fmt --check (per package) pass (Rust 1.99.1, macOS)
  • New or changed behaviour / the fixed bug has a unit test: elements::tests::web_contents_keep_screen_coordinates (tree shape taken from the real page: NavigationBar, WebView at y=100, inner node at (0,0) with the WebView's size, text at y=375). It fails on main with (59,488) and passes with the fix at (59,388)
  • No existing test removed, ignored or loosened
  • #[ignore] tests of the changed area run by hand: none apply

2. Real-device regression

  • Device, iOS version, host (Linux / macOS): iPhone 17 (iPhone18,3), iOS 26.5.2, macOS (Apple silicon); WDA 16.14.0 started in the background by the daemon
  • tests/device/run.py default steps pass on this branch (exit code 0): 14 calls, 0 failed, 10.7 s
  • The same steps on the latest release, same device, for comparison; no step clearly slower: v0.2.7 release binary, 14 calls, 0 failed, 11.7 s. (The first 0.2.7 attempt hit the known half-dead WDA of case feat(viewer): the page's requests share a connection #103 at the second Home press, the daemon restarted WDA, and the re-run below passed; not related to this change.)
  • Steps exercising this change: in an app's in-app browser page (WebView below a 44 pt navigation bar), iphone_list_elements with a filter on two headings, then iphone_tap_element on a heading and on a link; compared each position with the screenshot
  • Longer run: not affected (element parsing only)
run.py output
this branch:

=== iphone_status {} 0.19s
WDA 就绪: True
锁屏: 未锁屏
前台 App: com.apple.springboard
屏幕: 402x874 点
守护进程: 就绪
=== iphone_tap_element {"text": "不存在的按钮xyz"} 1.01s
没找到包含 "不存在的按钮xyz" 的元素,先用 iphone_list_elements 或截图看看屏幕
=== iphone_press_button {"button": "home"} 1.02s
已按 home
=== iphone_launch_app {"app": "时钟"} 1.41s
已启动 com.apple.mobiletimer
=== iphone_list_elements {"limit": 12} 0.17s
StaticText "0秒钟" id="stopwatch-time" @(201,258) 402x107
Image "秒表" @(62,452) 94x94
Button "分段" @(62,452) 92x92
Image "秒表" @(340,452) 94x94
Button "启动" @(340,452) 92x92
PageIndicator "第1页(共2页), 数码秒表" value="第1页(共2页), 数码秒表" @(201,454) 58x33
Table "空列表" @(201,694) 402x361
TabBar "标签页栏" @(201,832) 402x83
Button "世界时钟" @(74,822) 98x54
Button "闹钟" @(158,822) 99x54
Button "秒表" value="1" @(244,822) 99x54
Button "计时器" @(328,822) 98x54
=== iphone_tap_element {"text": "秒表", "then": "elements", "then_filter": "Button"} 0.85s
已点击 Image "秒表" @(62,452) 94x94(共匹配 3 个,点的是第 0 个)
界面已稳定(0.3 秒);但界面和操作前一样:操作可能没生效(点在了空白处、被别的东西挡住,或 App 还没反应过来),先截图看清再决定,不要原样再做一遍
--- 操作后的界面 ---
Button "分段" @(62,452) 92x92
Button "启动" @(340,452) 92x92
Button "世界时钟" @(74,822) 98x54
Button "闹钟" @(158,822) 99x54
Button "秒表" value="1" @(244,822) 99x54
Button "计时器" @(328,822) 98x54
=== iphone_screenshot {} 0.20s
402x874 点,图像像素即点坐标(经 WDA)
=== iphone_screenshot {"x": 0, "y": 0, "width": 430, "height": 60} 0.14s
区域 (0,0) 起 402x60 点,放大 2.99 倍;图中像素 (u,v) 对应屏幕点 (0+u/2.99, 0+v/2.99)(经 WDA)
=== iphone_press_button {"button": "home"} 1.58s
已按 home
=== iphone_swipe {"from_x": 215, "from_y": 400, "to_x": 215, "to_y": 700} 1.63s
已滑动 (215,400) → (215,700)
=== iphone_type_text {"text": "设置"} 0.87s
已输入 2 个字符
=== iphone_wait_for {"text": "设置", "timeout": 5} 0.08s
已出现 Cell "设置" id="Identifier:GridCell,Section:0,Row:0" @(201,158) 366x115(0.1 秒)
=== iphone_screenshot {} 0.36s
402x874 点,图像像素即点坐标(经 WDA)
=== iphone_press_button {"button": "home"} 1.11s
已按 home
--- 14 calls, 0 failed, 10.7s in all

latest release (v0.2.7):

=== iphone_status {} 0.07s
WDA 就绪: True
锁屏: 未锁屏
前台 App: com.apple.mobiletimer
屏幕: 402x874 点
守护进程: 就绪
=== iphone_tap_element {"text": "不存在的按钮xyz"} 0.23s
没找到包含 "不存在的按钮xyz" 的元素,先用 iphone_list_elements 或截图看看屏幕
=== iphone_press_button {"button": "home"} 1.65s
已按 home
=== iphone_launch_app {"app": "时钟"} 0.85s
已启动 com.apple.mobiletimer
=== iphone_list_elements {"limit": 12} 0.18s
StaticText "0秒钟" id="stopwatch-time" @(201,258) 402x107
Image "秒表" @(62,452) 94x94
Button "分段" @(62,452) 92x92
Image "秒表" @(340,452) 94x94
Button "启动" @(340,452) 92x92
PageIndicator "第1页(共2页), 数码秒表" value="第1页(共2页), 数码秒表" @(201,454) 58x33
Table "空列表" @(201,694) 402x361
TabBar "标签页栏" @(201,832) 402x83
Button "世界时钟" @(74,822) 98x54
Button "闹钟" @(158,822) 99x54
Button "秒表" value="1" @(244,822) 99x54
Button "计时器" @(328,822) 98x54
=== iphone_tap_element {"text": "秒表", "then": "elements", "then_filter": "Button"} 0.84s
已点击 Image "秒表" @(62,452) 94x94(共匹配 3 个,点的是第 0 个)
界面已稳定(0.2 秒);但界面和操作前一样:操作可能没生效(点在了空白处、被别的东西挡住,或 App 还没反应过来),先截图看清再决定,不要原样再做一遍
--- 操作后的界面 ---
Button "分段" @(62,452) 92x92
Button "启动" @(340,452) 92x92
Button "世界时钟" @(74,822) 98x54
Button "闹钟" @(158,822) 99x54
Button "秒表" value="1" @(244,822) 99x54
Button "计时器" @(328,822) 98x54
=== iphone_screenshot {} 0.21s
402x874 点,图像像素即点坐标(经 WDA)
=== iphone_screenshot {"x": 0, "y": 0, "width": 430, "height": 60} 0.12s
区域 (0,0) 起 402x60 点,放大 2.99 倍;图中像素 (u,v) 对应屏幕点 (0+u/2.99, 0+v/2.99)(经 WDA)
=== iphone_press_button {"button": "home"} 1.82s
已按 home
=== iphone_swipe {"from_x": 215, "from_y": 400, "to_x": 215, "to_y": 700} 1.57s
已滑动 (215,400) → (215,700)
=== iphone_type_text {"text": "设置"} 1.24s
已输入 2 个字符
=== iphone_wait_for {"text": "设置", "timeout": 5} 0.13s
已出现 Cell "设置" id="Identifier:GridCell,Section:0,Row:0" @(201,158) 366x115(0.1 秒)
=== iphone_screenshot {} 0.45s
402x874 点,图像像素即点坐标(经 WDA)
=== iphone_press_button {"button": "home"} 2.26s
已按 home
--- 14 calls, 0 failed, 11.7s in all

3. Regression cases (docs/dev/regression-cases.md)

  • All cases still hold; device cases in the changed area re-run: none of the existing device cases cover web content
  • A newly fixed problem is added as a row: #145

4. Real-device proof of the fix or feature

  • Device, iOS version, host (Linux / macOS): iPhone 17, iOS 26.5.2, macOS
  • Bug fix: reproduced on the release, gone with this branch, same device and steps
  • Untested conditions: landscape; iPad; iOS 27 native mode (this path only runs on WDA's XML)
Steps and results (before / after)

Same in-app browser page, same scroll position. On the screenshot the first heading's text is centred at about (59,388) and the second at about (59,595). Page texts are replaced with placeholders.

v0.2.7:
  iphone_list_elements {"filter": "<heading 1>"}
    Other "<heading 1>" value="1" @(201,494) 364x45
    StaticText "<heading 1>" @(59,488) 80x27        <- 100 pt below where it is
  iphone_list_elements {"filter": "<heading 2>"}
    StaticText "<heading 2>" @(59,694) 80x27        <- on screen at ~595
  iphone_tap_element {"text": "<heading 1>"}
    已点击 Other "<heading 1>" value="1" @(201,494)  -> the tap landed 100 pt low and opened the first article listed under the heading

this branch:
  iphone_list_elements {"filter": "<heading 1>"}
    Other "<heading 1>" value="1" @(201,394) 364x45
    StaticText "<heading 1>" @(59,388) 80x27        <- matches the screenshot
  iphone_list_elements {"filter": "<heading 2>"}
    StaticText "<heading 2>" @(59,594) 80x27
  iphone_tap_element {"text": "<link>"}
    已点击 Link "<link>" @(77,679) 82x22            -> that link's page opened

Checklist

  • MCP tool names, parameters and default outputs stay compatible (docs/decisions.md D10)
  • New or changed dependencies are pinned to exact versions; vendor changes recorded in THIRD_PARTY.md: no dependency changes

🤖 Generated with Claude Code

The widget heuristic in parse_tree treats a node at (0,0) as the start of
relative coordinates and adds its parent's position to everything below.
Inside a WKWebView there is a node that reports (0,0) at the WebView's
own size, while the web elements under it use screen coordinates. With
the WebView below a navigation bar (an app's in-app browser, y=100) every
element came out 100 pt too low, and tapping by text hit the next row.
Safari's WebView starts at y=0, so it never showed there.

A WebView and everything inside it are now left out of the heuristic.
Regression case #145; unit test web_contents_keep_screen_coordinates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant