MOB-99: retry-harden iOS accessibility-tree hit-testing - #84
Merged
Conversation
nif_tap_xy's simulator branch and nif_ax_action_at_xy both did a single find_a11y_at_point pass and gave up with :no_element_at_point on the first miss. SwiftUI's accessibility tree can lag a layout or navigation pass by a run loop tick or more — a synthetic tap issued the instant a screen mounts (the common automated-test pattern) can race it, even though the element's *frame* is already correct by then (tracked separately via MobFrameTracker's GeometryReader callback, unaffected by the same lag). Extracted find_a11y_at_point_in_windows_retrying/1: up to 4 attempts, 50ms apart. The retry sleep happens on the calling (NIF) thread between dispatch_sync calls, never inside the main-thread block itself — sleeping there would block the very run loop SwiftUI needs to finish building the tree, guaranteeing the wait never resolves. Also corrects CLAUDE.md's iOS accessibility activation section: the TODO asking for mix mob.connect to run VoiceOver activation automatically was stale — mob_dev's MobDev.Connector.connect_all/1 already does this (companion fix in mob_dev adds the settle delay the same doc already called for but the code never implemented).
…stants From code review on PR #84: - nif_tap_xy's simulator branch found the element via the retry helper, then did a SEPARATE dispatch_sync re-scanning all windows/scenes from scratch for the text-field focus walk. With an overlapping window (e.g. a keyboard window), that independent re-scan could resolve a DIFFERENT window than the one the element was actually found in. Restructured so find-then-act happens atomically inside one dispatch_sync per retry attempt via a new mob_retry_main_thread_bool helper — same window used throughout, and the common (no-retry-needed) case is back to one dispatch_sync round-trip instead of two. - nif_ax_action_at_xy had the same find/act split, with the retry sleep gap between them (up to 150ms) as a real TOCTOU window — a recycled table/collection view cell could receive an action meant for a different row while still returning :ok. New mob_retry_main_thread_found_action helper makes find+act atomic here too, while still distinguishing "not found" (retried) from "found but action unsupported/failed" (a stable outcome — retrying it wouldn't help, so it stops immediately rather than burning through all attempts). - nif_long_press_xy's accessibility fallback still called the old single-shot find_a11y_at_point with no retry — the exact race this whole PR targets was still open there. Wrapped its already-atomic find+act body in the same retry helper. - Extracted the duplicated scene/window-enumeration loop (introduced by the original PR's split) into one shared find_a11y_at_point_in_current_windows helper, used by all three call sites — no more copy-pasted loop to keep in sync. - Retry count/delay are now named constants (kA11yLookupMaxAttempts, kA11yLookupRetryDelay) with a comment clarifying they're unrelated to mob_dev's separate ~500ms post-connect settle delay, not the same figure wearing two names. - Both retry helpers now log which attempt succeeded (or that all attempts failed) — closes the "genuine miss vs retry-exhaustion both look identical" debuggability gap the review flagged, without changing the Erlang-visible return shape. Not addressed (documented, not silently dropped): tap_xy/ax_action_at_xy remain regular (non-dirty) NIFs, which can now block a scheduler thread for up to ~150-200ms on a full retry exhaustion. This mirrors an existing, already-documented tradeoff for this whole test-harness file (see the comment at the NIF registration table) rather than a new regression — moving these to dirty NIFs is a bigger call belonging to that existing decision, not this PR. find_a11y_by_label (used by tap-by-label, a different code path never reported as broken) got no retry treatment either — same reasoning, left as a known follow-up if it turns out to need it. Device-verified on a real iOS simulator: tap_id (:ok, focus confirmed), ax_action_at_xy (both distinct outcomes: :action_failed when found vs. :no_element_at_point when not), and long_press_xy (:ok fallback, screen process stays alive) all still work correctly after the refactor.
GenericJam
added a commit
that referenced
this pull request
Aug 26, 2026
Ships MOB-98 (native component events arriving as charlists, not binaries) and MOB-99 (iOS accessibility-tree hit-testing race in Mob.Test.tap_id/2 and friends), both merged from PR #83 and #84 with their review-fix follow-ups. Also fixes an intermittent test flake in component_test.exs surfaced by this release's preflight — the other half of a Mob.ComponentRegistry named-GenServer race MOB-98 already covered on the component_server_test.exs side. See CHANGELOG.md for the full breakdown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigated the reported gap:
Mob.Test.frame(node, id)returns the correct frame for a tagged element, butMob.Test.tap_id(node, id)returns{:error, :no_element_at_point}for some SwiftUI native views. Confirmed this isMob.Test's own in-process hit-testing mechanism, not ordinary touchscreen interaction (idb-driven raw taps at the same coordinates work fine — verified separately during MOB-98 device acceptance).Root cause:
frame()is driven byMobFrameTracker'sGeometryReadercallback (mob_register_frame), which fires as soon as SwiftUI computes layout.tap_id()→tap_xy()walks the accessibility tree (find_a11y_at_pointinios/mob_nif.m) via a completely separate mechanism that SwiftUI populates lazily — the two aren't guaranteed to settle at the same moment, so a tap issued immediately after a screen mounts or navigates (the common automated-test pattern) can race the accessibility tree, even though the frame is already correct.find_a11y_at_point_in_windows_retrying/1: retries the point lookup up to 4 times, 50ms apart, before giving up. The sleep happens on the calling NIF thread betweendispatch_synccalls — never inside the main-thread block itself, since blocking there would freeze the very run loop SwiftUI needs to finish building the tree.nif_tap_xy's simulator branch andnif_ax_action_at_xy(the two direct callers of this exact pattern) to use it.CLAUDE.md's iOS accessibility activation section — its TODO asking formix mob.connectto auto-activate VoiceOver was stale;mob_devalready does this. Companion PR (mob_dev#46) adds the settle delay the doc calls for but the code never actually implemented.Test plan
mix test— 1015 passed (no Elixir changes in this repo's fix, ObjC + doc only)mix format/mix credo --strict/clang-format/swiftlint— clean (1 pre-existing unrelatedforce_castwarning, untouched)mix mob.connectpipeline (companion mob_dev fix):Mob.Test.frame/2+Mob.Test.tap_id/2called with zero artificial delay immediately after connect, on a:text_fieldwith an:id— succeeded and the field became first responder (confirmed viafocused: truein the live socket assigns)Honest caveat: I could not cleanly reproduce a hard, reliable failure on my available simulators — this Mac's simulators appear to retain accessibility-service state across restarts in a way that made a genuinely cold-start repro difficult to isolate (likely residual from heavy
idb/accessibility-tooling usage earlier in the session). The fix addresses the real, provable gap in the mechanism (retry-hardening + honoring the documented settle delay) rather than a confirmed single root cause. If this doesn't fully resolve it for the original reporter's exact component tree, the next step needs their real reproduction.Linear: MOB-99 (companion PR: mob_dev fix/mob-99-ios-native-view-hit-testing)
🤖 Generated with Claude Code