Let apps mark the views that take direct touch

[masonasons]

Direct touch today is all or nothing for an app: when it's on, the whole screen goes to the app. That suits a game, but not an app where only part of a screen wants raw touch. FastPlay, an audio player, has a touch pad in its player (swipes seek, two-finger swipes adjust effects, a two-finger tap plays and pauses), while the rest of the app (its lists, buttons and settings) needs explore by touch. On iOS, VoiceOver handles this per view with "allows direct interaction"; Android has nothing an app can use, since only an accessibility service can set the passthrough region.

This lets an app mark the views that take touches directly, and Backtalk passes through only those.

What an app does

Declare the capability on <application> or the launch <activity>:

<meta-data
    android:name="dev.nvgt.capability.DIRECT_TOUCH_REGIONS"
    android:value="true" />

declare the marking ID in its resources:

<item name="dev_nvgt_direct_touch_region" type="id" />

and give it to each view that should take raw touch:

gesturePad.id = R.id.dev_nvgt_direct_touch_region

Several views can share it. Backtalk turns a region app on the first time it sees it, as it does for DIRECT_TOUCH apps, and the user can still turn it off in the Apps list. The constants are in DirectTouchSettings next to CAPABILITY_KEY.

How it behaves

• Backtalk finds the marked views with findAccessibilityNodeInfosByViewId("<package>:id/dev_nvgt_direct_touch_region") on each of the app's application windows. The app searches its own views to answer, so a look is one call per window however big the app's tree is, and there's no walk and no node limit. Each window's root is fetched once while the window stays.
• Only an app that declares regions and is turned on is ever asked, and only while nothing else (the shade, another app, a focused text field, the screen being off) keeps its direct touch off.
• Its content changes and scrolls are looked at no more than once every 150 ms, throttled rather than debounced, so an app that updates often (a player's position) still gets looked at. While the region is showing, the first change after a quiet spell is looked at at once, so a marked view going away hands touch back without waiting.
• The passthrough region is the on-screen bounds of the visible marked views, clipped to their window, minus every window layered above (the app's own dialogs and popups, other apps) and minus the windows whole-screen mode already leaves out (system windows, accessibility overlays, the keyboard unless direct typing is on). The navigation bar (with lift to activate on it) and keyboards in the direct touch list are added back as in whole-screen mode.
• "Direct touch on" and "Direct touch off", and their vibrations, are said when the app may have direct touch and when it no longer may, as for a game. The region going empty and back, as a marked view scrolls away or another screen of the app shows, isn't announced.
• A dialog from the same app doesn't turn region mode off (the region just leaves the dialog out); the screen turning off, the shade, another app's window and a focused text field still do. In DirectTouchPolicy this is one new input, regionMode.
• The manifest is still read at an app's first sight, as master does since Look up windows less often when the screen changes #164, and otherwise only for an app in front that is turned on, again only after another app's was read.
• Apps with plain DIRECT_TOUCH, or turned on by the user, keep whole-screen behaviour exactly as before. An app that declares both gets region mode.

differences.md has a new section, "Direct touch on part of the screen", for users and app developers.

Testing

• gradlew :talkback:testPhoneDebugUnitTest --tests 'com.google.android.accessibility.talkback.directtouch.*': 53 tests pass. DirectTouchControllerTest has eight new tests on top of the ones from Look up windows less often when the screen changes #164, which pass unchanged: the region for a marked view, the region coming and going without announcements, the throttle under a stream of content changes (8 looks for 100 changes over a second), the first change being looked at at once, other apps and a region app that's turned off never being asked, the app's dialog being cut out, and the screen turning off. They count the asks for marked views, root fetches and manifest reads.
• assemblePhoneDebug and :lintPhoneDebug pass.
• Not yet tried on a device, and not on a watch. Worth checking there: the app answering the ID lookup for its views, the window layer order of an app's popups, and a marked view's bounds while scrolled. FastPlay's Android app (masonasons/FastPlay, android/) declares the capability and marks its touch pad with the ID, so it can serve as a test app.

🤖 Generated with Claude Code

https://claude.ai/code/session_019iUrBiG422jeRdCb9hkkS9

[trypsynth]

Thanks, this is a nice extension, and the trust model is fine: region apps are turned on at first sight just like DIRECT_TOUCH apps, the user can still turn them off, and it reuses the keyboard direct touch code. A few things before merging:

• Main-thread cost. Finding marked views walks the app's tree on the main thread, up to 500 getChild/extras calls, each a call into the app, every 150 ms while the app is in front and sending content changes. That happens even when nothing is marked and direct touch is off. We only recently got rid of this kind of scan because it made watches lag. Could it skip the walk entirely while nothing is marked, or only rescan on a signal from the app, and be tried on a device, ideally a watch?
• Announcement chatter. "Direct touch on/off" now follows the region going empty and non-empty, so a marked pad scrolling in and out of view, or appearing on a screen change, will toggle announcements and vibrations often. Please announce only when the app's direct touch actually starts or ends.
• No tests yet for the throttle and the controller wiring; the pure-logic tests are good.
• Overlap with Look up windows less often when the screen changes #164. It conflicts with Look up windows less often when the screen changes #164, which refactors the same controller and policy, so one of them will need a rebase.

[masonasons]

Thanks for the review. I've pushed changes for each point:

• Main-thread cost. The walk is gone. Apps now mark views with the view ID dev_nvgt_direct_touch_region instead of the node extra, and Backtalk asks each of the app's windows for findAccessibilityNodeInfosByViewId("<package>:id/dev_nvgt_direct_touch_region"). The app searches its own views to answer, so a look is one call per window (usually one) however big the tree is, and each window's root is fetched once while it stays. Only an app that declares regions and is turned on is asked, and only while nothing else keeps its direct touch off; other apps are never asked. The controller test counts this: 100 content changes over a second make 8 lookups and no new root fetches. I haven't tried it on a device yet, and I don't have a watch, so that part is still open.
• Announcement chatter. "Direct touch on" and "off" now follow whether the app may have direct touch (it comes to the front, the shade opens, a text field takes focus, the screen turns off), not the region going empty and back. A marked pad scrolling away or another screen of the app showing is silent.
• Tests. DirectTouchControllerTest has eight new tests for the throttle, the first change being looked at at once, the region coming and going without announcements, the app's dialog being cut out, apps that are never asked, and the screen turning off. The tests from Look up windows less often when the screen changes #164 pass unchanged.
• Look up windows less often when the screen changes #164. It was already merged, so I merged master in and redid the controller and policy changes on top of it, keeping its lazy root checks and first-sight manifest reads.

The PR description is updated to match.

[trypsynth]

Thanks, this addresses everything I asked: one lookup per window instead of the tree walk, announcements only when direct touch really starts or ends, tests for the throttle and the controller, and it's merged with master after #164. I've approved CI.

Before merging, could you try it on a phone yourself, with an app that marks views, and say what you tried and what happened? I'll test it too, but I'd like to know the code I'm running has been tried by a person, not only by the tests. A watch would be a bonus if you can get hold of one, but a phone is what I'm asking for and a watch is not at all a merge requirement

[masonasons]

Tried it on a phone: a Galaxy S23+ on Android 16, running a debug build of this branch (b59baff, with today's master merged in) as the screen reader, and FastPlay's Android app, which declares DIRECT_TOUCH_REGIONS and gives its player's touch pad the ID dev_nvgt_direct_touch_region.

It worked as intended:

• Direct touch took effect only on the touch pad: swiping there went straight to FastPlay, and the rest of the app stayed with explore by touch.
• "Direct touch on" and "Direct touch off" were said only when FastPlay came to the front and when it was left, not when moving onto or off the touch pad.

I haven't tried a watch.