fix: rescale gesture coordinates for compatibility-mode window mismatches

[mykola-mokhnach]

Summary

Fixes appium/appium#16185: taps/gestures can land on the wrong point when an app's own window size doesn't match the device's (e.g. an iPhone-only app running in iPad compatibility mode, or any other compatibility-scaled window).

Root cause

XCUIAutomation.framework: XCUICoordinate built from an element/app origin plus a raw points offset (coordinateWithOffset:) is never rescaled by XCTest — the offset is added as-is and handed straight to event synthesis. The only place XCTest does correct for a mismatched window is when resolving a normalized offset (coordinateWithNormalizedOffset:), which is recomputed against the element's live frame at synthesis time. That's also why [element tap] (used by WDA's plain /click) already worked correctly — it resolves via the AX-server-computed activationPoint, not client-side math — while touch/perform, W3C actions, force touch, drag, and scroll (which all built raw offsets from a WDA-reported rect) did not.

Fix

Normalize every gesture offset against the target element's own frame before building the coordinate, via a new shared FBCoordinateWithAnchorOffset() helper (FBMathUtils), instead of composing a raw points offset. Applied to tap/touch/W3C actions, force touch, drag/swipe, and scroll (incl. scroll-to-visible, which now anchors to a live-resolved element rather than the application and surfaces failures instead of silently retrying with an unscaled coordinate).

One case is intentionally left as a documented, self-verifying limitation: an element-less absolute offset (e.g. a raw viewport x,y with no target element) has no frame to normalize against and XCTest has no mechanism to correct it — covered by a test wrapped in XCTExpectFailure that will start failing loudly if XCTest ever fixes this itself.

[mykola-mokhnach]

@Dan-Maor Could you please help verifying it on a real device?

[Dan-Maor]

I’m out of the country at the moment, I’ll be back on Tuesday and will be happy to help testing it then.

[mykola-mokhnach]

Thanks for the extra normal-window coverage — that matches the fix pushed in 2169f81 (switched FBCoordinateWithAnchorOffset to element.wdFrame, same change validated in your experiment), along with a reworked test that asserts the exact landing position. Agreed the actual compatibility-mode mismatch still needs real-hardware verification; that's pending via @Dan-Maor.

[KazuCocoa]

Let me check with the latest.
Yes, his double check will also help. I tested real iphone se2 and ipad air 11 inch, so it makes sense to get more screen coverage he has

[KazuCocoa]

Retested exact commit 2169f8181f7538a506581c41a6e403f3db516cf8 on the physical devices. The previously reported denominator error is fixed, and the new regression test catches the old behavior.

Configuration	Coordinate checks	Scrolling
iPhone-only probe on iPad, 375 × 667 app / 820 × 1180 SpringBoard	All 13 center/element-relative checks pass; the documented absolute viewport case still fails	Scroll-down and row 60 pass
Full-screen native iPad probe, 820 × 1180	14/14 pass	Both pass
iPhone SE, 375 × 667	14/14 pass	Both pass

For the compatibility-mode probe, the W3C offset error fell from 46.75 to 0.13 app points, the element-relative tap error from 132.82 to 0.07, and the drag endpoint error from 121.91 to 0.15. These match the earlier local experiment. Raw viewport error remains 218.91 points, as documented. Native-iPad errors were zero; the iPhone's maximum was 0.71 points from rounding its half-point viewport center to integer x/y. Element-relative iPhone taps/drags were accurate within floating-point precision.

I also ran IntegrationTests_3/FBElementOffsetTapTest/testTapWithElementOffsetUnderWindowSizeMismatch directly against the stock iPhone-only IntegrationApp on the iPad:

• 2169f81: passed, one test executed, zero failures, not skipped.
• Negative control: same commit and unchanged test, with only the helper's denominator reverted from element.wdFrame to element.frame. After a clean app install, it failed at the exact-location assertion in FBTapTest.m:183.

The first control attempt failed on a post-gesture touch-view lookup; I repeated it cleanly to confirm the intended coordinate assertion failure. For direct XCTest execution, the generated integration runner needed its already-built WebDriverAgentLib.framework embedded and the runner re-signed before it could launch. No test-source changes were needed.

Environment: iPad14,8 / iPadOS 26.6.1 / Xcode 26.6 (17F113), and iPhone12,8 / iOS 27.0 / Xcode 27 beta (27A5252f); Appium 3.5.2 / XCUITest driver 12.8.2. WDA was rebuilt from the requested commit for each toolchain and its running build identity checked.

Both findings from my earlier review are addressed by this commit in these tests. Coverage is still portrait-only for the listed gestures; landscape, force touch, picker wheels, the historical banking app, and the full repository suite were not tested.

[KazuCocoa]

lgtm, great. will post the result with ipad mini as well later.

I'll add landscape mode as well. Noticed that current coverage is only portrait

[KazuCocoa]

Additional device coverage for 2169f8181f7538a506581c41a6e403f3db516cf8: iPad mini (iPad16,2), iPadOS 26.6.1.

• iPhone compatibility mode: app 375 × 667, SpringBoard 744 × 1133. All element-offset taps and drag endpoints passed, with maximum error 0.23 app points across the runs. The fresh coordinate run passed all 13 center/element-relative checks. Absolute viewport coordinates still failed as documented (185.98-point error).
• Native full-screen iPad mode: app and SpringBoard both 744 × 1133. All 14 coordinate checks passed. Center/element-relative taps and drag had zero measured error; the viewport center differed by 0.5 points due to integer rounding.
• Scroll-down and scroll-to-row-60 passed in both modes.

One observation from the initial compatibility run: a native center-click command returned without error but did not increment the probe's touch count. The old coordinates were still present, so I counted it as an unrecorded touch, not a misplaced one. Twelve targeted native-click repetitions then passed with immediate measurements, and a fresh-session repeat of the full coordinate sequence also passed all 13 center/element-relative checks. The initial event remains unexplained and is retained as an intermittent observation; I have not attributed it to the PR.

Same probe and 2-app-point tolerance as before, using Xcode 26.6 (17F113), Appium 3.5.2 / XCUITest driver 12.8.2. WDA was rebuilt for this device from the exact commit and its running identity checked. These were portrait probe tests; the stock XCTest regression was not rerun on the mini.

[KazuCocoa]

Follow-up on 2169f8181f7538a506581c41a6e403f3db516cf8: I repeated the iPad mini (iPad16,2, iPadOS 26.6.1) checks with the device in landscape. Center/element-relative coordinates still work, but mobile: scroll fails in both native and compatibility modes.

Configuration	Coordinate result	Scroll-down / scroll-to-row-60
Native full-screen, 1133 × 744	14/14 passed; element-relative errors 0, viewport error 0.5 points from integer rounding	No content movement; row-60 command failed and row 60 was not displayed
iPhone compatibility on landscape device	13/13 center/element-relative checks passed; maximum error 0.226 app points	No content movement; row-60 command timed out at the client's 90-second limit and row 60 was not displayed

The compatibility app retained a 375 × 667 portrait UIKit layout on the landscape device. WDA reported its window as 667 × 375 and SpringBoard as 1133 × 744. This distinction matters: the test covers that actual window/orientation mismatch. The unsupported viewport tap recorded no new touch; its stored coordinates remained from the previous drag, so they should not be interpreted as a measured viewport miss distance.

For the scroll failure, the native table rect was (20,214,1093,490) and compatibility table rect (20,202,335,420). Both used:

mobile('scroll', {'elementId': table, 'direction': 'down', 'distance': 0.5})
mobile('scroll', {'elementId': table, 'name': 'probe-row-60'})

XCTest logged these normalized table gestures:

Native:        [0.13, 1.28] -> [0.13, 0.16]
Compatibility: [-0.35, 0.75] -> [-0.35, 0.35]

Both start outside the table. The native run ended in the app switcher; its row-60 command reported Failed to find scrollable visible parent with 2 visible children. After explicitly reactivating the probe, native scroll-down at distances 0.5 and 0.25 still left content Y at 0. Compatibility also remained at content Y 0 while the row-60 command repeatedly attempted the off-table gesture.

As a control, direct W3C element-relative drags scrolled the same table in both modes: compatibility content Y 0 → 198.5; native full-screen 364 → 730.5. The native control's nonzero starting offset came from an earlier drag while the reinstalled app was temporarily windowed; it was maximized and its 1133 × 744 geometry verified before the full-screen control.

The leading explanation is a coordinate-space mismatch between self.scrollingFrame (snapshot visibleFrame) and anchorElement.frame in fb_scrollAncestorScrollViewByVectorWithinScrollViewFrame:anchorElement:error:. Those frames need to be expressed in the same coordinate space before calculating the normalized offsets. The off-table logs support investigating that conversion, but I have not validated a specific fix or established that changing only the denominator would suffice.

These are results on the requested PR commit, using the same signed WDA/probe artifacts as the earlier mini portrait run (Appium 3.5.2, XCUITest driver 12.8.2, Xcode 26.6). Portrait scrolling passed previously. A base-revision landscape comparison has not been run, so I am reporting a failure on this head without claiming it is proven to be newly introduced. The element-offset fix remains effective in this landscape setup; the scroll path needs further investigation.

[mykola-mokhnach]

Thanks for tracking this down. The scrollingFrame (snapshot visibleFrame) vs anchorElement.frame (live) mismatch in fb_scrollAncestorScrollViewByVectorWithinScrollViewFrame:anchorElement:error: does look like a real, separate bug — but since it also reproduces in the native (non-mismatched) landscape case, a plain wdFrame swap wouldn't fully explain or fix it, and it's outside what this PR set out to address (element-relative tap/drag offsets). Scoping it out of this PR and tracking it as a follow-up rather than blocking this fix on it.

[KazuCocoa]

Yeah, all of my tests in both orientation had good results. lgtm overall