Skip to content

display(iOS): fix touch gestures and add drag to scroll mode - #7893

Merged
osy merged 6 commits into
mainfrom
ios-touch-gestures
Sep 21, 2026
Merged

osy merged 6 commits into
mainfrom
ios-touch-gestures

Conversation

@osy

@osy osy commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

This is a rewrite of #7697 by @ifilipis. That PR identified real problems with touch input on iOS, but it arrived as one 900-line commit that also changed defaults and switched existing gestures off for current users. This branch keeps the fixes, splits them into one commit per issue, and leaves the existing touch modes and all defaults as they are.

What changes

  1. Long press works after half a second. The long press waited for the tap recognizer to fail, which only happens after the finger has been down for 0.75 to 1.5 s depending on the iOS version. A normal long press came out as a tap, so "Right Click" produced a left click or nothing and "Click & Hold" never pressed the button.
  2. Pans start at once. Every pan waited for the three finger keyboard swipes to fail, which takes up to half a second. Moving the cursor with one finger or scrolling with two did nothing for that long, and a short two finger scroll was mostly lost. The pinch no longer waits either; instead it may not begin while three fingers are down, because before iOS 18 it begins with two of three fingers and takes the touches from the keyboard swipe (that also stops three fingers from zooming on those versions). When two finger pan and two finger swipe both scroll the wheel, they now run together and the swipe follows "Invert Scroll".
  3. Touches that start at a screen edge reach the guest. A drag from the top edge pulled down Notification Center and the guest saw nothing. Uses SwiftUI's defersSystemGestures(on:) (iOS 16+) next to the existing persistentSystemOverlays(.hidden). On iOS 18 and later a swipe up from the bottom edge still goes home, because the system ignores the deferral while the home indicator is hidden.
  4. Middle click and right/middle drag. "Middle Click" for long press and two finger tap, "Right Click & Hold" and "Middle Click & Hold" for the two and three finger pans. New values for existing settings; defaults unchanged.
  5. New "Touch mode (drag to scroll)". In the existing touch modes the left button goes down the moment a finger lands, so every two finger tap, scroll or pinch starts with a left press and a short drag at the first finger (selected text, stray marks, accidental clicks). The new mode never presses on touch down: a tap clicks, one finger drags the content, a long press runs the long press action, and a long press that moves becomes a left button drag. A ring shows when the long press is recognized. A quick second tap close to the first clicks on the same spot so the guest sees a double click.
  6. A cancelled drag releases its button. When the system takes over a touch that is holding a button (for example a long press drag that turns into the swipe to go home), the guest was left with the button held down.

Claims in #7697

# Claim in #7697 Here Notes Simulator Tested on device
1 Conflicts everywhere: scroll triggers pinch and taps Partly The stray left press is gone in the new mode; the old touch modes keep it by design, since there the touch is the drag. "Scroll triggers pinch" did not reproduce. Yes Yes
2 Gesture types very limited Yes Middle click, right and middle hold. Yes Yes
3 Scroll only triggers after dragging across half the screen Yes First scroll event from 0.48 s to 0.17 s, first cursor motion from 0.52 s to 0.07 s, in every mode. Yes Yes
4 Undocumented, unconfigurable gestures No The three finger keyboard swipe and pinch-with-Move-Screen still have no setting. The PR's settings defaulted to Disabled, which turned both gestures off for existing users. n/a n/a
5 Old touch mode conserved, new mode added Yes The old modes are untouched here. In #7697 they were not: pinch and the keyboard swipe were off by default and the per-swipe wheel stopped working. Yes Yes
6 Tap / click works the same way Yes Plus the double tap fix above. Yes Yes
7 One finger drag scrolls, no erroneous clicks Yes Content follows the finger (the PR scrolled the other way), and a touch stops a fling. Yes Yes
8 Long press right click with a visual indicator Partly Indicator included, and the underlying bug is fixed in every mode. The default stays "Click & Hold". Yes Yes
9 Long press + drag does a left drag Yes Without the PR's extra setting. Yes Yes
10 Two finger tap clicks at the first touch No Still clicks midway between the fingers, as on main. n/a n/a
11 Two finger drag resolves conflicts, no delay Mostly Starts at UIKit's normal 10 pt threshold. A multi finger drag still moves the cursor to the point between the fingers. Yes Yes
12 Two finger swipe resolved correctly, fine-tunable Differently Keeps UISwipeGestureRecognizer; pan and swipe run together when both scroll. The PR's velocity based detection was left out. Yes Yes
13 Pinch based on absolute finger travel Left out A 10 pt finger drift during a pan does not start a pinch, so there was nothing to fix. Yes Yes
14 Pinch is two fingers only Yes Needed before iOS 18, where three fingers zoomed; see change 2. Yes Yes
15 Same fixes for three finger gestures, swipe in settings Partly The three finger pan gets the new actions but still waits for the keyboard swipe (about 0.3 s), since both use three touches. No three finger tap or swipe settings. Yes Yes
16 Gestures persist once one has fired UIKit Left to the recognizers' own exclusion. Cannot be simulated (see Testing). No Yes
17 New mouse events: left, right and middle drag plus clicks Yes See change 4, plus change 6. Yes Yes
18 Fixed gestures near the screen edges Yes With the SwiftUI modifier instead of patching UIViewController; the patched getter was ignored on iOS 17. Yes Yes
19 Defaults changed to match MS RDP Left out Would change behaviour for every user who never opened Settings. n/a n/a
20 Nothing in the driver changed Same Only Platform/iOS. n/a n/a

Resolves #4738

Testing

Testing: Tested by a human on iPhone 16 Pro, iOS 26.7. The author acknowledges that this change has been tested and/or reviewed by a human in accordance with UTM's AI contribution guidelines.

The device test covered 22 steps over four settings configurations: all defaults in Drag cursor mode, the new mode with the new actions, the old touch mode with pan and swipe bound to different actions, and long press disabled. It included the cases that cannot be simulated: three finger swipes and two finger gestures with the fingers landing at different times. One step was skipped (Apple Pencil with palm rejection, and an external pointer, while the new mode is selected). The first device pass found that double tap did not work in the new mode; that is fixed in change 5 and was retested.

Automated testing, in addition: every change was run before and after on iPhone simulators with iOS 17.0, 18.0, 26.0 and 27.0, against a Windows XP guest. A UI test bundle injected synthetic multi finger touches, and a local logging shim (not part of this PR) recorded the mouse events sent to the guest and the recognizer state changes. iOS 18, 26 and 27 agree on every check. iOS 17 differs, which is where the pinch guard in change 2 and the SwiftUI modifier in change 3 came from. Synthetic touches land all fingers at the same instant, so staggered landing was only tested on the device.

Not tested: iOS 15 and 16 (on iOS 15 change 3 has no effect, the modifier needs iOS 16), iPad, and visionOS (the modifier is unavailable there and compiled out; the visionOS target was not built).

🤖 Generated with Claude Code

osy added 6 commits September 20, 2026 04:29
The long press recognizer required the tap recognizers to fail first. A
tap only fails once the finger has been held for well over a second, so
a normal long press was delivered as a tap when the finger lifted: the
"Right Click" action produced a left click (or nothing in touch mode)
and "Click & Hold" never pressed the button.

Reverse the dependency so the tap waits for the long press, which fails
as soon as a short touch ends and so adds no delay to taps. When the
long press action is disabled the recognizer is not allowed to begin,
otherwise holding before lifting would no longer click.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
Every pan and the pinch required the three finger keyboard swipes to
fail first, and a swipe recognizer only fails once the touch has moved
far enough to be judged or after about half a second. Moving the cursor
with one finger, scrolling with two or pinching therefore did nothing
for a quarter to half a second, and a short or slow two finger scroll
was mostly lost.

A swipe can only be confused with a pan that uses the same number of
touches, so only the three finger pan still waits for the keyboard
swipes. The pinch needs a guard instead: before iOS 18 it begins with
two of three fingers and would take the touches from the swipe, so it is
not allowed to begin while three fingers are down. That also stops three
fingers from zooming the display on those versions.

The two finger pan still waits for the two finger swipe when the two are
bound to different actions. When both scroll the mouse wheel there is
nothing to tell apart: the pan scrolls right away and a swipe adds its
wheel tick on top, where before any steady two finger drag was taken for
a swipe and scrolled a single tick. Because the two now scroll together,
the swipe follows the "Invert Scroll" setting like the pan.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
A touch that starts at the top or bottom edge of the screen is held
back by the system in case it turns into a system gesture. A drag from
the top edge of the VM display pulled down Notification Center and the
guest never saw the touch at all.

Ask the system to defer its edge gestures while the VM display is up,
next to where the home indicator is already hidden, so the first swipe
goes to the guest and only a second one is taken by the system. This
uses the SwiftUI modifier, which needs iOS 16. Overriding the view
controller preference instead needs the hosting controller patched to
forward it, and was ignored on iOS 17.

On iOS 18 and later a swipe up from the bottom edge still goes home,
because the system ignores the deferral there while the home indicator
is hidden. Touches at the bottom edge do reach the guest immediately.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
Touch gestures could only produce a left drag and a right click, so a
guest that needs the middle button (paste or pan in many Linux and CAD
programs) or a right button drag could not be used without a mouse.

Add "Middle Click" to the long press and two finger tap actions, and
"Right Click & Hold" and "Middle Click & Hold" to the two and three
finger pan actions. These are new values for existing settings and the
defaults are unchanged.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
In the touch modes the left button goes down the moment a finger lands,
because the touch itself is the drag. Every other gesture starts with a
finger landing too, so a two finger tap, scroll or pinch first clicked
and dragged whatever was under the first finger until the gesture was
recognized, and a long press held the left button for half a second
before its right click.

Add "Touch mode (drag to scroll)" as a third touch mode that behaves
like a touch screen and never presses a button on touch down. A tap is
a left click, a long press runs the long press action, and dragging one
finger scrolls the mouse wheel so that the content follows the finger.
Since a touch no longer drags, a long press that moves turns into a left
button drag so windows and selections can still be moved with one
finger. A ring is shown when the long press is recognized because
nothing in the guest reacts until the finger lifts.

A finger never lands on the same spot twice, and a guest only counts two
clicks on the same spot as a double click, so a second tap that follows
quickly and close by clicks where the first one did.

The existing touch modes are unchanged.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
A gesture that holds a mouse button only released it when the gesture
ended. When the system takes over the touch instead, for example a long
press drag that starts at the bottom edge and turns into the swipe to go
home, the gesture is cancelled and the guest was left with the button
held down until the next click.

Reworked from #7697.

Assisted-by: Claude:claude-fable-5-1
@osy osy added this to the v5.0 milestone Sep 21, 2026
@osy
osy merged commit 00e83da into main Sep 21, 2026
54 checks passed
@osy
osy deleted the ios-touch-gestures branch September 22, 2026 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Touch screen mode optimization suggestions

1 participant