iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour

AGENTS.md

@@ -76,6 +76,19 @@ Mob.Test.inspect(node)           # full snapshot: screen, assigns, nav stack, wi
 This is faster, exact (not pixel-inferred), and works without taking a
 screenshot. Use it as the default.
 
+**Colour is the exception.** `view_tree/1`'s `bg_color`/`text_color` are `nil`
+for virtually all SwiftUI content (iOS 26 paints via `SDFLayer` or rasterises
+into `contents` — colour for 4 of 443 nodes when measured). To verify what was
+actually drawn, sample pixels:
+
+```elixir
+Mob.Test.sample_color(node, "my-card")   # {:ok, %{average: 0xFF2196F3, dominant: ..., ...}}
+```
+
+It crops in the NIF, so only that element's pixels cross dist. See
+`decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md` and
+`decisions/2026-08-10-sample-region-crops-natively-and-stays-debug-only.md`.
+
 ### Drive
 
 ```elixir
@@ -202,6 +215,17 @@ These are the things we've burned ourselves on. Following them isn't optional.
     rather than duplicating them. See
     `mob_dev/decisions/2026-06-19-mob-adopt-lives-in-mob_dev.md`.
 
+14. **Don't drive iOS by coordinate — use `Mob.Test.tap/2`.** `tap_xy/3` works
+    on the simulator only for elements SwiftUI gives an accessibility action
+    (`Button`, text fields); a `Box` with `on_tap:` has none. On a physical
+    device the injected IOHID touch is accepted and never delivered, so every
+    coordinate fails. Both now return `{:error, :no_effect}` instead of the
+    `:ok` they used to — a harness that reported success for taps that did
+    nothing let a downstream iOS renderer bug sit unverified for weeks. When
+    you add a harness NIF, make its success value mean *observed effect*, never
+    *the platform API didn't complain*. See
+    `decisions/2026-08-09-tap-xy-reports-observed-effect.md`.
+
 ## Where to look
 
 | Question | File |

CLAUDE.md

@@ -170,14 +170,24 @@ over Erlang distribution to a remote test runner or agent.
   `[{type, label, value, {x,y,w,h}}, ...]` tuples. Works on any app with zero modification.
 - `ui_debug/0` — raw accessibility dump for debugging
 
-**Phase 2 — Synthetic interaction (complete)**
+**Phase 2 — Synthetic interaction (partial — see the honesty note below)**
 
 - `tap/1` — tap by accessibility label
 - `tap_xy/2` — tap at screen coordinates (with responder-chain walk to focus text fields)
 - `type_text/1` — type into the focused text field
 - `delete_backward/0`, `key_press/1`, `clear_text/0` — keyboard control
 - `long_press_xy/3`, `swipe_xy/4` — gesture synthesis
 
+Coordinate-driven input is *not* finished, whatever the list above implies.
+`tap_xy/2` now returns `:ok` only when the app demonstrably reacted; on the
+simulator that limits it to `Button`s and text fields, and on a physical device
+the injected IOHID touch is accepted but never delivered, so every coordinate
+returns `{:error, :no_effect}`. `swipe_xy/4` and `long_press_xy/3` still report
+on acceptance and their `:ok` is unverified. Drive Mob screens with
+`Mob.Test.tap/2` (by tag). See
+`decisions/2026-08-09-tap-xy-reports-observed-effect.md` and
+`decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md`.
+
 **Phase 3 — Full cocoon / event interception (future)**
 
 Intercept the touch event stream before it reaches the app's responder chain. The BEAM

android/jni/mob_nif.zig

@@ -499,10 +499,17 @@ export fn nif_ui_tree(
 // nif_ui_view_tree/0 — returns nested-map UI tree from MobBridge.uiViewTree().
 //
 // Bridge contract: Kotlin returns a JSON string of the form
-//   {"type":"root","label":null,"value":null,"frame":[0,0,W,H],"children":[...]}
+//   {"type":"root","label":null,"value":null,"frame":[0,0,W,H],
+//    "bg_color":null,"text_color":null,"children":[...]}
 // parsed by Mob.Test.tree/1 (jason decode is fast; no need for a C-side
 // JSON tokenizer). Returns {:error, :not_loaded} when MobBridge.uiViewTree()
 // isn't present (early-adopter apps without registry).
+//
+// bg_color/text_color are 0xAARRGGBB integers (guides/theming.md), matching
+// what iOS build_view_node emits, so Mob.Test.view_tree/1 reads the same on
+// both platforms. null when the view paints no colour of its own. No shipped
+// MobBridge.kt implements uiViewTree() yet — Android view_tree is
+// {:error, :not_loaded} until one does; keep these keys when it lands.
 export fn nif_ui_view_tree(
     env: ?*erts.ErlNifEnv,
     argc: c_int,

decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md

@@ -0,0 +1,91 @@
+# iOS physical-device IOHID tap injection is accepted but never delivered
+
+- Date: 2026-08-09
+- Status: proposed (investigation; no fix in this change)
+
+## Context
+
+Driving a physical iPhone (iOS 26.5.2) over dist, `Mob.Test.tap_xy/3` returned
+`:ok` for every coordinate and nothing ever happened on screen. Timeboxed
+investigation of why, recorded so the next attempt doesn't restart from zero.
+The honesty fix that makes this visible instead of silent is
+`2026-08-09-tap-xy-reports-observed-effect.md`; this file is about the
+underlying delivery failure.
+
+## What the code does today
+
+`mob_send_touch_phase` (`ios/mob_nif.m`) on iOS 26+:
+
+1. `dlsym(RTLD_DEFAULT, "IOHIDEventCreateDigitizerFingerEvent")`
+2. builds a **bare finger event** with normalized coords
+   (`pt / UIScreen.mainScreen.bounds.size`), `eventMask = Range|Touch|Position`,
+   `tipPressure` 1.0/0.0, `range`/`touch` booleans keyed to the phase
+3. `((HandleFn)objc_msgSend)(app, @selector(_handleHIDEvent:), hidEvent)`
+4. reads back `[app _touchesEvent]` and, *if* `allTouches.count > 0`, calls
+   `[window sendEvent:ev]`
+5. **returns `YES` unconditionally**
+
+## Findings
+
+1. **The return value never described delivery.** Step 5 returns `YES` whenever
+   the symbol resolved and the selector exists. Step 4 already contains the
+   evidence of failure — `allTouches.count` is 0 — and the code logs it, then
+   ignores it. This is the whole reason the false `:ok` reached callers. Cheapest
+   possible next step: return `NO` when `allTouches.count == 0` after
+   `_handleHIDEvent:`. That converts a silent lie into a typed error with no new
+   API surface, and gives a device-side signal to iterate against.
+
+2. **A bare finger event is the wrong shape.** UIKit's HID path expects a *parent*
+   `kIOHIDEventTypeDigitizer` event with finger events appended as children
+   (`IOHIDEventCreateDigitizerEvent` + `IOHIDEventAppendEvent`), not a lone
+   finger event. Injecting the child directly is the most likely reason UIKit
+   ingests it and produces no `UITouch`. This is the shape every working
+   out-of-process touch injector uses.
+
+3. **Sender/context routing is unset.** `_handleHIDEvent:` routes by the window's
+   context id. Nothing in the current path sets a sender id on the event
+   (`IOHIDEventSetSenderID`). The `window_info` diagnostic already in
+   `nif_tap_xy` — it probes `_contextId` / `_windowContextID` / `contextId` /
+   `_displayID` on the key window — is a leftover from a previous run at this
+   and is exactly the value that would need to be attached.
+
+4. **The manual `UITouch` path may be revivable on iOS 26.** The file's own
+   runtime enumeration (dated 2026-04-21, in the comment above the private
+   category declarations) records that iOS 26 *promoted* `setWindow:`,
+   `setView:`, `setPhase:`, `setTimestamp:` and `setTapCount:` to public, leaving
+   only `_setLocationInWindow:resetPrevious:` private, and that `_touchesEvent`
+   still exists on `UIApplication`. The `#if` structure sends iOS 26 down the
+   HID path unconditionally, so the promoted-setter variant of the `< 26` path
+   has never actually been tried on 26. It is a small, self-contained experiment.
+
+## Concrete next steps, cheapest first
+
+1. Make `mob_send_touch_phase` return `NO` when no `UITouch` materialised
+   (`[app _touchesEvent].allTouches.count == 0`). Pure honesty, no risk.
+2. Try the promoted-public-setter `UITouch` + `[window sendEvent:]` path on
+   iOS 26 (finding 4). Guard with `respondsToSelector:` as the existing code
+   does; fall through to HID if it doesn't take.
+3. If 2 fails, build the parent digitizer event and append the finger
+   (finding 2), and set the sender id from the window context id the
+   `window_info` probe already retrieves (finding 3).
+4. Orthogonal and probably the highest value for Mob apps specifically: make
+   coordinate taps unnecessary. `MobFrameTracker` in `ios/MobRootView.swift`
+   already reports frames for nodes carrying an `:id`; extending it to nodes
+   carrying `on_tap` would let `tap_xy` resolve a point to a tap handle and call
+   `mob_send_tap` directly — the same thing SwiftUI's `onTapGesture` does, with
+   no private API at all. Deliberately not done here: another agent is working
+   in `MobRootView.swift`.
+5. Also for the simulator: adding `.accessibilityAddTraits(.isButton)` +
+   `.accessibilityAction { tap() }` alongside the `.onTapGesture` in
+   `MobRootView.swift` would make `accessibilityActivate` genuinely fire
+   `on_tap`, fixing simulator coordinate taps for `Box`/`Row`/`Column`. Same
+   file, same reason for deferring.
+
+## Consequences
+
+- Coordinate-driven taps on a physical iPhone are **not working** and are now
+  reported as `{:error, :no_effect}` rather than `:ok`. `Mob.Test.tap/2` (by tag)
+  is unaffected and remains the supported way to drive Mob screens.
+- None of the above is verified on hardware from this change — it is a reading
+  of the code plus the reported device behaviour. Step 1 is the instrumentation
+  that would turn it into evidence.

decisions/2026-08-09-tap-xy-reports-observed-effect.md

@@ -0,0 +1,76 @@
+# tap_xy reports observed effect, not API acceptance
+
+- Date: 2026-08-09
+- Status: accepted
+
+## Context
+
+`mob_nif:tap_xy/2` returned `ok` whenever the platform input mechanism did not
+raise an error. That is not the same as "the tap worked", and the gap is wide:
+
+- **iOS simulator** — the branch activates the accessibility element under the
+  point. SwiftUI only maps `accessibilityActivate` to a default action for
+  `Button`-like views. Mob's `Box` / `Row` / `Column` implement `on_tap:` with a
+  plain `.onTapGesture`, which has no accessibility action, so activation is
+  accepted, the handler never runs, and `tap_xy` still answered `ok`.
+- **iOS physical device** (iPhone, iOS 26.5.2) — the branch synthesises an
+  `IOHIDEvent` and asks `mob_send_touch_phase` whether it worked.
+  `mob_send_touch_phase` returns `YES` as soon as
+  `IOHIDEventCreateDigitizerFingerEvent` resolves and `_handleHIDEvent:` exists,
+  i.e. it reports *API availability*, never delivery. Every coordinate returned
+  `ok`, including coordinates with nothing tappable under them, and nothing
+  happened for any of them.
+
+The damage is not the broken tap — it's the false `ok`. An agent or a test
+driving a device cannot distinguish a working tap from a no-op, so failures read
+as passes. An iOS renderer bug sat unverified in a downstream repo for weeks
+because of exactly this.
+
+## Decision
+
+`ok` now means **the app demonstrably reacted**, on both iOS paths.
+
+A process-wide counter (`g_ui_event_seq` in `ios/mob_nif.m`) is bumped by every
+send helper that routes a user-originated event into the BEAM — `mob_send_tap`,
+`mob_send_event` (focus / blur / submit / select) and `mob_send_change`.
+`nif_tap_xy` samples it before injecting, then polls for up to
+`MOB_TAP_SETTLE_MS` (300ms) after. It also hit-tests up front on *both* paths;
+previously only the device path did.
+
+Return contract:
+
+| Value | Meaning |
+|---|---|
+| `ok` | An event reached the BEAM within 300ms. |
+| `{error, no_view_at_point}` | Hit-test found nothing — outside every visible window. |
+| `{error, no_element_at_point}` | Simulator: a view is there, no AX element to activate. |
+| `{error, no_effect}` | The OS accepted the input; no handler ran. |
+| `{error, Probe}` | Device: the private injection API is missing. |
+
+`tap_xy` moves to `ERL_NIF_DIRTY_JOB_IO_BOUND` — it can now block for the settle
+window, which is far too long for a normal scheduler. (It already slept 100ms on
+the device path, so this also fixes a pre-existing scheduler violation.)
+
+Chose observation over "just document the limitation" (option b in the brief)
+because a counter is cheap, needs no Swift changes, and keeps `ok` meaningful if
+and when the device path is repaired — the honest answer changes automatically
+rather than needing a doc edit.
+
+## Consequences
+
+- The platform matrix in `Mob.Test` gets worse on paper and honest in practice:
+  simulator `tap_xy` works for `Button` and text fields only; device `tap_xy`
+  returns `{error, no_effect}` for every coordinate today. `Mob.Test.tap/2` (by
+  tag) remains the way to drive Mob screens and is unaffected.
+- **Sidecar mode caveat.** The counter only sees handlers Mob owns. Driving a
+  non-Mob app, a genuinely successful tap still reports `{error, no_effect}`
+  because there is nothing to observe. Documented on `Mob.Test.tap_xy/3`;
+  callers there should verify with `ui_tree/1` or a screenshot. Closing this
+  properly needs Phase 3 event interception (see `CLAUDE.md`), where the BEAM
+  sees the touch stream itself.
+- `swipe_xy/4` and `long_press_xy/3` share the injection path and still report on
+  acceptance. Flagged in the matrix as unverified; converting them is the same
+  mechanical change and is deliberately left out of this diff.
+- 300ms is a guess tuned to SwiftUI's tap-gesture recognition delay. A slow
+  handler that only sends to the BEAM after heavy work would report `no_effect`.
+  Raise the constant if that shows up; don't remove the check.

decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md

@@ -0,0 +1,59 @@
+# view_tree colour cannot be read from the layer tree on iOS 26
+
+- Date: 2026-08-09
+- Status: accepted
+
+## Context
+
+`Mob.Test.view_tree/1` returned no colour, so a styling regression was invisible
+to the one introspection API meant to show what the device actually drew. The
+motivating bug: under a glass theme the iOS renderer discarded every Box
+background colour, and confirming it required pixel-diffing screenshots by hand.
+
+The first fix read `UIView.backgroundColor` / `UILabel.textColor`. On a real
+Mob screen that produced colour for **2 of 443 nodes**. A second attempt walked
+the layer tree as well — `CALayer.backgroundColor`, `CAShapeLayer.fillColor`,
+`CATextLayer.foregroundColor` — reaching **4 of 443**.
+
+`:mob_nif.ui_paint_debug/0` (added here) censuses where paint actually lives.
+Measured on an iPhone 17 simulator, iOS 26, against a 61-component app:
+
+    total_views=442  groups=18
+    204x SwiftUI._UIInheritedView / CALayer            all paint props 0
+     72x UIPlatformGlassInteractionView / CALayer      all paint props 0
+     64x SwiftUI._UIInheritedView / SwiftUI.SDFLayer   all paint props 0
+     37x _UIInheritedView / SDFLayer + SDFPortalLayer  all paint props 0
+     14x _UIInheritedView / CGDrawingLayer             all 0, contents=14
+
+Every group reports zero for view background, layer background, shape fill,
+gradient, text-layer foreground and UIKit text colour.
+
+## Decision
+
+Layer-based colour extraction is a dead end for SwiftUI-rendered content, not a
+bug to keep fixing. On iOS 26 SwiftUI paints through `SDFLayer` (a
+signed-distance-field renderer whose colour is not exposed as a layer property)
+or rasterises into `contents`. Mob's renderer uses `.background(Color, in:)` and
+`.foregroundColor` for every Box and Text, so essentially all app content is
+invisible to this approach.
+
+The colour fields stay, because they are correct where they resolve (UIKit
+chrome, the root background, and any future Android implementation), and the
+`:class` field added alongside them tells the caller which renderer drew a node
+and therefore why a colour is `nil`. But they must not be presented as a way to
+verify Mob styling.
+
+**Colour verification belongs on screenshot sampling.** `Mob.Test` already has
+`screenshot/2` and `element_frames/1`; sampling a node's frame from a capture
+answers "what colour was actually drawn" without depending on private renderer
+internals, works identically on both platforms, and survives Apple changing them.
+
+## Consequences
+
+- `view_tree` colour is best-effort and nearly always `nil` for SwiftUI content.
+  Documented as such in `Mob.Test`.
+- A styling regression is still NOT detectable from the tree. The follow-up —
+  a sampling helper (e.g. `sample_color(node, id)`) — is what closes that gap.
+- `ui_paint_debug/0` is kept deliberately. It answered this question in one RPC
+  and will answer it again when Apple changes the internals; re-deriving it by
+  reading SwiftUI class names across device rebuilds is the slow way.