diff --git a/AGENTS.md b/AGENTS.md index dd86082..f37cf07 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,6 +76,19 @@ Mob.Test.inspect(node) # full snapshot: screen, assigns, nav stack, wi This is faster, exact (not pixel-inferred), and works without taking a screenshot. Use it as the default. +**Colour is the exception.** `view_tree/1`'s `bg_color`/`text_color` are `nil` +for virtually all SwiftUI content (iOS 26 paints via `SDFLayer` or rasterises +into `contents` — colour for 4 of 443 nodes when measured). To verify what was +actually drawn, sample pixels: + +```elixir +Mob.Test.sample_color(node, "my-card") # {:ok, %{average: 0xFF2196F3, dominant: ..., ...}} +``` + +It crops in the NIF, so only that element's pixels cross dist. See +`decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md` and +`decisions/2026-08-10-sample-region-crops-natively-and-stays-debug-only.md`. + ### Drive ```elixir @@ -202,6 +215,17 @@ These are the things we've burned ourselves on. Following them isn't optional. rather than duplicating them. See `mob_dev/decisions/2026-06-19-mob-adopt-lives-in-mob_dev.md`. +14. **Don't drive iOS by coordinate — use `Mob.Test.tap/2`.** `tap_xy/3` works + on the simulator only for elements SwiftUI gives an accessibility action + (`Button`, text fields); a `Box` with `on_tap:` has none. On a physical + device the injected IOHID touch is accepted and never delivered, so every + coordinate fails. Both now return `{:error, :no_effect}` instead of the + `:ok` they used to — a harness that reported success for taps that did + nothing let a downstream iOS renderer bug sit unverified for weeks. When + you add a harness NIF, make its success value mean *observed effect*, never + *the platform API didn't complain*. See + `decisions/2026-08-09-tap-xy-reports-observed-effect.md`. + ## Where to look | Question | File | diff --git a/CLAUDE.md b/CLAUDE.md index 828bddd..3c8fa3c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -170,7 +170,7 @@ over Erlang distribution to a remote test runner or agent. `[{type, label, value, {x,y,w,h}}, ...]` tuples. Works on any app with zero modification. - `ui_debug/0` — raw accessibility dump for debugging -**Phase 2 — Synthetic interaction (complete)** +**Phase 2 — Synthetic interaction (partial — see the honesty note below)** - `tap/1` — tap by accessibility label - `tap_xy/2` — tap at screen coordinates (with responder-chain walk to focus text fields) @@ -178,6 +178,16 @@ over Erlang distribution to a remote test runner or agent. - `delete_backward/0`, `key_press/1`, `clear_text/0` — keyboard control - `long_press_xy/3`, `swipe_xy/4` — gesture synthesis +Coordinate-driven input is *not* finished, whatever the list above implies. +`tap_xy/2` now returns `:ok` only when the app demonstrably reacted; on the +simulator that limits it to `Button`s and text fields, and on a physical device +the injected IOHID touch is accepted but never delivered, so every coordinate +returns `{:error, :no_effect}`. `swipe_xy/4` and `long_press_xy/3` still report +on acceptance and their `:ok` is unverified. Drive Mob screens with +`Mob.Test.tap/2` (by tag). See +`decisions/2026-08-09-tap-xy-reports-observed-effect.md` and +`decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md`. + **Phase 3 — Full cocoon / event interception (future)** Intercept the touch event stream before it reaches the app's responder chain. The BEAM diff --git a/android/jni/mob_nif.zig b/android/jni/mob_nif.zig index 152bb3b..92055c2 100644 --- a/android/jni/mob_nif.zig +++ b/android/jni/mob_nif.zig @@ -499,10 +499,17 @@ export fn nif_ui_tree( // nif_ui_view_tree/0 — returns nested-map UI tree from MobBridge.uiViewTree(). // // Bridge contract: Kotlin returns a JSON string of the form -// {"type":"root","label":null,"value":null,"frame":[0,0,W,H],"children":[...]} +// {"type":"root","label":null,"value":null,"frame":[0,0,W,H], +// "bg_color":null,"text_color":null,"children":[...]} // parsed by Mob.Test.tree/1 (jason decode is fast; no need for a C-side // JSON tokenizer). Returns {:error, :not_loaded} when MobBridge.uiViewTree() // isn't present (early-adopter apps without registry). +// +// bg_color/text_color are 0xAARRGGBB integers (guides/theming.md), matching +// what iOS build_view_node emits, so Mob.Test.view_tree/1 reads the same on +// both platforms. null when the view paints no colour of its own. No shipped +// MobBridge.kt implements uiViewTree() yet — Android view_tree is +// {:error, :not_loaded} until one does; keep these keys when it lands. export fn nif_ui_view_tree( env: ?*erts.ErlNifEnv, argc: c_int, diff --git a/decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md b/decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md new file mode 100644 index 0000000..1cf4def --- /dev/null +++ b/decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md @@ -0,0 +1,91 @@ +# iOS physical-device IOHID tap injection is accepted but never delivered + +- Date: 2026-08-09 +- Status: proposed (investigation; no fix in this change) + +## Context + +Driving a physical iPhone (iOS 26.5.2) over dist, `Mob.Test.tap_xy/3` returned +`:ok` for every coordinate and nothing ever happened on screen. Timeboxed +investigation of why, recorded so the next attempt doesn't restart from zero. +The honesty fix that makes this visible instead of silent is +`2026-08-09-tap-xy-reports-observed-effect.md`; this file is about the +underlying delivery failure. + +## What the code does today + +`mob_send_touch_phase` (`ios/mob_nif.m`) on iOS 26+: + +1. `dlsym(RTLD_DEFAULT, "IOHIDEventCreateDigitizerFingerEvent")` +2. builds a **bare finger event** with normalized coords + (`pt / UIScreen.mainScreen.bounds.size`), `eventMask = Range|Touch|Position`, + `tipPressure` 1.0/0.0, `range`/`touch` booleans keyed to the phase +3. `((HandleFn)objc_msgSend)(app, @selector(_handleHIDEvent:), hidEvent)` +4. reads back `[app _touchesEvent]` and, *if* `allTouches.count > 0`, calls + `[window sendEvent:ev]` +5. **returns `YES` unconditionally** + +## Findings + +1. **The return value never described delivery.** Step 5 returns `YES` whenever + the symbol resolved and the selector exists. Step 4 already contains the + evidence of failure — `allTouches.count` is 0 — and the code logs it, then + ignores it. This is the whole reason the false `:ok` reached callers. Cheapest + possible next step: return `NO` when `allTouches.count == 0` after + `_handleHIDEvent:`. That converts a silent lie into a typed error with no new + API surface, and gives a device-side signal to iterate against. + +2. **A bare finger event is the wrong shape.** UIKit's HID path expects a *parent* + `kIOHIDEventTypeDigitizer` event with finger events appended as children + (`IOHIDEventCreateDigitizerEvent` + `IOHIDEventAppendEvent`), not a lone + finger event. Injecting the child directly is the most likely reason UIKit + ingests it and produces no `UITouch`. This is the shape every working + out-of-process touch injector uses. + +3. **Sender/context routing is unset.** `_handleHIDEvent:` routes by the window's + context id. Nothing in the current path sets a sender id on the event + (`IOHIDEventSetSenderID`). The `window_info` diagnostic already in + `nif_tap_xy` — it probes `_contextId` / `_windowContextID` / `contextId` / + `_displayID` on the key window — is a leftover from a previous run at this + and is exactly the value that would need to be attached. + +4. **The manual `UITouch` path may be revivable on iOS 26.** The file's own + runtime enumeration (dated 2026-04-21, in the comment above the private + category declarations) records that iOS 26 *promoted* `setWindow:`, + `setView:`, `setPhase:`, `setTimestamp:` and `setTapCount:` to public, leaving + only `_setLocationInWindow:resetPrevious:` private, and that `_touchesEvent` + still exists on `UIApplication`. The `#if` structure sends iOS 26 down the + HID path unconditionally, so the promoted-setter variant of the `< 26` path + has never actually been tried on 26. It is a small, self-contained experiment. + +## Concrete next steps, cheapest first + +1. Make `mob_send_touch_phase` return `NO` when no `UITouch` materialised + (`[app _touchesEvent].allTouches.count == 0`). Pure honesty, no risk. +2. Try the promoted-public-setter `UITouch` + `[window sendEvent:]` path on + iOS 26 (finding 4). Guard with `respondsToSelector:` as the existing code + does; fall through to HID if it doesn't take. +3. If 2 fails, build the parent digitizer event and append the finger + (finding 2), and set the sender id from the window context id the + `window_info` probe already retrieves (finding 3). +4. Orthogonal and probably the highest value for Mob apps specifically: make + coordinate taps unnecessary. `MobFrameTracker` in `ios/MobRootView.swift` + already reports frames for nodes carrying an `:id`; extending it to nodes + carrying `on_tap` would let `tap_xy` resolve a point to a tap handle and call + `mob_send_tap` directly — the same thing SwiftUI's `onTapGesture` does, with + no private API at all. Deliberately not done here: another agent is working + in `MobRootView.swift`. +5. Also for the simulator: adding `.accessibilityAddTraits(.isButton)` + + `.accessibilityAction { tap() }` alongside the `.onTapGesture` in + `MobRootView.swift` would make `accessibilityActivate` genuinely fire + `on_tap`, fixing simulator coordinate taps for `Box`/`Row`/`Column`. Same + file, same reason for deferring. + +## Consequences + +- Coordinate-driven taps on a physical iPhone are **not working** and are now + reported as `{:error, :no_effect}` rather than `:ok`. `Mob.Test.tap/2` (by tag) + is unaffected and remains the supported way to drive Mob screens. +- None of the above is verified on hardware from this change — it is a reading + of the code plus the reported device behaviour. Step 1 is the instrumentation + that would turn it into evidence. diff --git a/decisions/2026-08-09-tap-xy-reports-observed-effect.md b/decisions/2026-08-09-tap-xy-reports-observed-effect.md new file mode 100644 index 0000000..61bf0ca --- /dev/null +++ b/decisions/2026-08-09-tap-xy-reports-observed-effect.md @@ -0,0 +1,76 @@ +# tap_xy reports observed effect, not API acceptance + +- Date: 2026-08-09 +- Status: accepted + +## Context + +`mob_nif:tap_xy/2` returned `ok` whenever the platform input mechanism did not +raise an error. That is not the same as "the tap worked", and the gap is wide: + +- **iOS simulator** — the branch activates the accessibility element under the + point. SwiftUI only maps `accessibilityActivate` to a default action for + `Button`-like views. Mob's `Box` / `Row` / `Column` implement `on_tap:` with a + plain `.onTapGesture`, which has no accessibility action, so activation is + accepted, the handler never runs, and `tap_xy` still answered `ok`. +- **iOS physical device** (iPhone, iOS 26.5.2) — the branch synthesises an + `IOHIDEvent` and asks `mob_send_touch_phase` whether it worked. + `mob_send_touch_phase` returns `YES` as soon as + `IOHIDEventCreateDigitizerFingerEvent` resolves and `_handleHIDEvent:` exists, + i.e. it reports *API availability*, never delivery. Every coordinate returned + `ok`, including coordinates with nothing tappable under them, and nothing + happened for any of them. + +The damage is not the broken tap — it's the false `ok`. An agent or a test +driving a device cannot distinguish a working tap from a no-op, so failures read +as passes. An iOS renderer bug sat unverified in a downstream repo for weeks +because of exactly this. + +## Decision + +`ok` now means **the app demonstrably reacted**, on both iOS paths. + +A process-wide counter (`g_ui_event_seq` in `ios/mob_nif.m`) is bumped by every +send helper that routes a user-originated event into the BEAM — `mob_send_tap`, +`mob_send_event` (focus / blur / submit / select) and `mob_send_change`. +`nif_tap_xy` samples it before injecting, then polls for up to +`MOB_TAP_SETTLE_MS` (300ms) after. It also hit-tests up front on *both* paths; +previously only the device path did. + +Return contract: + +| Value | Meaning | +|---|---| +| `ok` | An event reached the BEAM within 300ms. | +| `{error, no_view_at_point}` | Hit-test found nothing — outside every visible window. | +| `{error, no_element_at_point}` | Simulator: a view is there, no AX element to activate. | +| `{error, no_effect}` | The OS accepted the input; no handler ran. | +| `{error, Probe}` | Device: the private injection API is missing. | + +`tap_xy` moves to `ERL_NIF_DIRTY_JOB_IO_BOUND` — it can now block for the settle +window, which is far too long for a normal scheduler. (It already slept 100ms on +the device path, so this also fixes a pre-existing scheduler violation.) + +Chose observation over "just document the limitation" (option b in the brief) +because a counter is cheap, needs no Swift changes, and keeps `ok` meaningful if +and when the device path is repaired — the honest answer changes automatically +rather than needing a doc edit. + +## Consequences + +- The platform matrix in `Mob.Test` gets worse on paper and honest in practice: + simulator `tap_xy` works for `Button` and text fields only; device `tap_xy` + returns `{error, no_effect}` for every coordinate today. `Mob.Test.tap/2` (by + tag) remains the way to drive Mob screens and is unaffected. +- **Sidecar mode caveat.** The counter only sees handlers Mob owns. Driving a + non-Mob app, a genuinely successful tap still reports `{error, no_effect}` + because there is nothing to observe. Documented on `Mob.Test.tap_xy/3`; + callers there should verify with `ui_tree/1` or a screenshot. Closing this + properly needs Phase 3 event interception (see `CLAUDE.md`), where the BEAM + sees the touch stream itself. +- `swipe_xy/4` and `long_press_xy/3` share the injection path and still report on + acceptance. Flagged in the matrix as unverified; converting them is the same + mechanical change and is deliberately left out of this diff. +- 300ms is a guess tuned to SwiftUI's tap-gesture recognition delay. A slow + handler that only sends to the BEAM after heavy work would report `no_effect`. + Raise the constant if that shows up; don't remove the check. diff --git a/decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md b/decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md new file mode 100644 index 0000000..179367e --- /dev/null +++ b/decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md @@ -0,0 +1,59 @@ +# view_tree colour cannot be read from the layer tree on iOS 26 + +- Date: 2026-08-09 +- Status: accepted + +## Context + +`Mob.Test.view_tree/1` returned no colour, so a styling regression was invisible +to the one introspection API meant to show what the device actually drew. The +motivating bug: under a glass theme the iOS renderer discarded every Box +background colour, and confirming it required pixel-diffing screenshots by hand. + +The first fix read `UIView.backgroundColor` / `UILabel.textColor`. On a real +Mob screen that produced colour for **2 of 443 nodes**. A second attempt walked +the layer tree as well — `CALayer.backgroundColor`, `CAShapeLayer.fillColor`, +`CATextLayer.foregroundColor` — reaching **4 of 443**. + +`:mob_nif.ui_paint_debug/0` (added here) censuses where paint actually lives. +Measured on an iPhone 17 simulator, iOS 26, against a 61-component app: + + total_views=442 groups=18 + 204x SwiftUI._UIInheritedView / CALayer all paint props 0 + 72x UIPlatformGlassInteractionView / CALayer all paint props 0 + 64x SwiftUI._UIInheritedView / SwiftUI.SDFLayer all paint props 0 + 37x _UIInheritedView / SDFLayer + SDFPortalLayer all paint props 0 + 14x _UIInheritedView / CGDrawingLayer all 0, contents=14 + +Every group reports zero for view background, layer background, shape fill, +gradient, text-layer foreground and UIKit text colour. + +## Decision + +Layer-based colour extraction is a dead end for SwiftUI-rendered content, not a +bug to keep fixing. On iOS 26 SwiftUI paints through `SDFLayer` (a +signed-distance-field renderer whose colour is not exposed as a layer property) +or rasterises into `contents`. Mob's renderer uses `.background(Color, in:)` and +`.foregroundColor` for every Box and Text, so essentially all app content is +invisible to this approach. + +The colour fields stay, because they are correct where they resolve (UIKit +chrome, the root background, and any future Android implementation), and the +`:class` field added alongside them tells the caller which renderer drew a node +and therefore why a colour is `nil`. But they must not be presented as a way to +verify Mob styling. + +**Colour verification belongs on screenshot sampling.** `Mob.Test` already has +`screenshot/2` and `element_frames/1`; sampling a node's frame from a capture +answers "what colour was actually drawn" without depending on private renderer +internals, works identically on both platforms, and survives Apple changing them. + +## Consequences + +- `view_tree` colour is best-effort and nearly always `nil` for SwiftUI content. + Documented as such in `Mob.Test`. +- A styling regression is still NOT detectable from the tree. The follow-up — + a sampling helper (e.g. `sample_color(node, id)`) — is what closes that gap. +- `ui_paint_debug/0` is kept deliberately. It answered this question in one RPC + and will answer it again when Apple changes the internals; re-deriving it by + reading SwiftUI class names across device rebuilds is the slow way. diff --git a/decisions/2026-08-10-sample-region-crops-natively-and-stays-debug-only.md b/decisions/2026-08-10-sample-region-crops-natively-and-stays-debug-only.md new file mode 100644 index 0000000..9bf6108 --- /dev/null +++ b/decisions/2026-08-10-sample-region-crops-natively-and-stays-debug-only.md @@ -0,0 +1,66 @@ +# Colour verification samples pixels: crop in the NIF, reduce in Elixir + +- Date: 2026-08-10 +- Status: accepted + +## Context + +`decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md` established +that no view- or layer-tree property can report the colour of SwiftUI-rendered +content on iOS 26, and named the follow-up: a sampling helper. The bug that +motivates it is a glass theme under which mob's iOS renderer discarded every Box +background, so `background: :primary` and `background: :surface_raised` rendered +identically — found only by pixel-diffing screenshots by hand. + +Three things had to be decided: where the crop happens, what a "colour" for a +region even is, and whether the capture primitive may ship in release builds. + +## Decision + +**Crop natively.** `:mob_nif.sample_region(X, Y, W, H)` renders the window with +the crop rect offset to the context origin, so only the region's pixels are +allocated and only they cross Erlang distribution. Returning a framebuffer and +cropping in Elixir would move megabytes per assertion. It reuses `screenshot/3`'s +capture path — both now call `mob_capture_window()` + `mob_capture_image()`, so +there is one window-picking and one rendering code path, not two. + +Coordinates are window points, the same space `element_frames/0` reports, so +`Mob.Test.sample_color(node, "my-card")` resolves a rect through `frame/2` with no +unit conversion. Rects are clamped to the window; a rect entirely outside it is +`{:error, :offscreen}`, never a plausible-looking black. + +**Reduce in pure Elixir, and report more than a mean.** `Mob.Test.reduce_rgba/3` +takes the buffer plus its dimensions and returns `%{average, dominant, +dominant_share, distinct, pixels}` — all colours `0xAARRGGBB`. A region is rarely +one flat colour (a card has text, a border, antialiased corners), so a bare mean +of a mostly-background region is misleading in both directions: it is *not* the +background, and it is not the text. `:dominant` (most frequent exact pixel, ties +broken toward the higher value so it is deterministic) is the background of a flat +fill; `:dominant_share` and `:distinct` tell the caller whether to believe it — +high share means a flat fill, low share means a gradient where only `:average` +carries meaning. Being pure and dimension-checked, it is unit-testable without a +device, and a buffer that doesn't match its declared size returns +`{:error, :size_mismatch}` rather than a colour derived from partial data. + +**`sample_region/4` stays `#if !MOB_RELEASE`** — *not* `#if !MOB_RELEASE || +defined(MOB_ENABLE_SCREENSHOT)` like its neighbour. It uses only public UIKit, so +App Store validation is not the constraint; the constraint is that arbitrary-rect +pixel reads reconstruct the screen region by region, which would silently give a +release build the capability `MOB_ENABLE_SCREENSHOT` exists to make a conscious +opt-in. Colour verification is a dev-time need — a shipped agent wants +`screenshot/3` to see the screen, not a sampling probe. `Mob.ReleaseScreenshotTest` +pins the guard so the distinction survives a future edit. + +## Consequences + +- A styling regression is now detectable programmatically: sample two Boxes and + compare `:dominant`. Equal samples for `:primary` vs `:surface_raised` reproduce + the original bug. +- iOS only. Android needs the same crop-in-the-render treatment against the + activity window; until then `sample_color/2` returns `{:error, {:badrpc, _}}` + there, and the platform matrix in `Mob.Test` says so. +- Payload is `w * h * screen_scale² * 4` bytes (a 100x50pt element ≈ 180 KB at + 3x). Sampling element frames is cheap; a full-screen rect is not, and the + docstring says not to. +- The reduction runs in the BEAM over a binary, so a very large region costs CPU + on the device. Same guidance: sample elements, not screens. diff --git a/ios/mob_nif.m b/ios/mob_nif.m index 23ab1d6..034c859 100644 --- a/ios/mob_nif.m +++ b/ios/mob_nif.m @@ -175,6 +175,29 @@ static void mob_handle_meta(int handle, uint64_t *seq_out, uint64_t *ts_out) { } static char g_transition[16] = "none"; +// ── UI-event observation counter (test-harness honesty) ────────────────────── +// +// Every user-originated event we hand to the BEAM bumps this counter. The +// synthetic-input NIFs sample it before and after injecting a touch so they can +// answer "did the app actually react?" instead of "did the injection API not +// return an error?". Without it tap_xy reports :ok whenever the private input +// API accepts the event — which on a physical device is always, even when the +// touch lands on nothing and no handler runs. +// +// Bumped from the send helpers rather than from the SwiftUI callbacks so it +// covers every route into the BEAM (tap, focus, blur, submit, select, change). +static _Atomic uint64_t g_ui_event_seq; + +#if !MOB_RELEASE // only the harness reads it; the writers stay unconditional +static uint64_t mob_ui_event_seq(void) { + return atomic_load_explicit(&g_ui_event_seq, memory_order_relaxed); +} +#endif + +static void mob_note_ui_event(void) { + atomic_fetch_add_explicit(&g_ui_event_seq, 1, memory_order_relaxed); +} + // Called from node onTap blocks — routes tap to BEAM via enif_send. static void mob_send_tap(int handle) { enif_mutex_lock(tap_mutex); @@ -186,6 +209,7 @@ static void mob_send_tap(int handle) { ERL_NIF_TERM tag = tap_handles[handle].tag; enif_mutex_unlock(tap_mutex); + mob_note_ui_event(); ErlNifEnv *msg_env = enif_alloc_env(); ERL_NIF_TERM msg = enif_make_tuple2(msg_env, enif_make_atom(msg_env, "tap"), enif_make_copy(msg_env, tag)); @@ -206,6 +230,7 @@ static void mob_send_event(int handle, const char *atom) { ERL_NIF_TERM tag = tap_handles[handle].tag; enif_mutex_unlock(tap_mutex); + mob_note_ui_event(); ErlNifEnv *msg_env = enif_alloc_env(); ERL_NIF_TERM msg = enif_make_tuple2(msg_env, enif_make_atom(msg_env, atom), enif_make_copy(msg_env, tag)); @@ -552,6 +577,7 @@ static void mob_send_change(int handle, ERL_NIF_TERM value_term) { ERL_NIF_TERM tag = tap_handles[handle].tag; enif_mutex_unlock(tap_mutex); + mob_note_ui_event(); ErlNifEnv *msg_env = enif_alloc_env(); ERL_NIF_TERM msg = enif_make_tuple3(msg_env, enif_make_atom(msg_env, "change"), enif_make_copy(msg_env, tag), @@ -3693,7 +3719,8 @@ static void ax_walk(void *elem, ErlNifEnv *env, ERL_NIF_TERM *list, int depth) { // - ui_tree returns a flat list of accessibility leaves; useful when an // agent wants the same view of the world VoiceOver/XCUITest see. // - ui_view_tree returns the full UIView hierarchy as a nested map, -// including non-accessible containers, with frames in window coords. +// including non-accessible containers, with frames in window coords and +// the colours actually painted (bg_color / text_color, 0xAARRGGBB). // Strict superset of what AX exposes — class names, hidden subviews, // things AX wouldn't surface. @@ -3738,6 +3765,142 @@ static void ax_walk(void *elem, ErlNifEnv *env, ERL_NIF_TERM *list, int depth) { return nil; } +// ── Drawn colour extraction for ui_view_tree ───────────────────────────────── +// +// Inverse of color_from_argb: packs a resolved UIColor into the repo's +// canonical 0xAARRGGBB integer (guides/theming.md — alpha first, NOT CSS +// #RRGGBBAA). Returns the atom nil when there is no colour, or when the colour +// is a pattern (or otherwise unconvertible) one with no single RGBA value. +// +// Must run on the main thread: dynamic colours (UIColor.labelColor, anything +// from an asset catalog) resolve against the current trait collection. +static ERL_NIF_TERM argb_term_from_uicolor(ErlNifEnv *env, UIColor *color) { + if (!color) + return enif_make_atom(env, "nil"); + CGFloat r = 0, g = 0, b = 0, a = 0; + if (![color getRed:&r green:&g blue:&b alpha:&a]) + return enif_make_atom(env, "nil"); + // Wide-gamut (Display P3) colours can land outside 0..1 once converted to + // sRGB components; clamp so the packed byte is always in range. + CGFloat comps[4] = {a, r, g, b}; + unsigned long argb = 0; + for (int i = 0; i < 4; i++) { + CGFloat c = comps[i] < 0 ? 0 : (comps[i] > 1 ? 1 : comps[i]); + argb = (argb << 8) | (unsigned long)(c * 255.0 + 0.5); + } + return enif_make_ulong(env, argb); +} + +// ── Where the paint actually lives ─────────────────────────────────────────── +// +// UIKit puts colour on the view (`UILabel.textColor`, `UIView.backgroundColor`). +// SwiftUI mostly doesn't: `.background(Color, in: shape)` and `.foregroundColor` +// (which is what `MobRootView` uses for every Box and Text) go through SwiftUI's +// own renderer, and the resulting colour lands on a CALayer — usually a +// CAShapeLayer fill — hanging off a structural view whose own backgroundColor +// stays nil. Reading only the view is why a 443-node dump came back with two +// colours, both system chrome. +// +// So each node harvests from its own layer subtree as well. A view's +// `layer.sublayers` includes its subviews' layers; those are excluded so a +// container never claims a child's paint as its own. + +// Depth cap: SwiftUI stacks a handful of layers per view, never dozens. Bounding +// the walk keeps ui_view_tree's cost linear in views, not in the whole layer graph. +#define MOB_LAYER_WALK_DEPTH 6 + +static BOOL mob_all_gradient_stops_equal(CAGradientLayer *gradient) { + if (gradient.colors.count < 2) + return YES; + id first = gradient.colors.firstObject; + for (id c in gradient.colors) { + if (!CGColorEqualToColor((__bridge CGColorRef)c, (__bridge CGColorRef)first)) + return NO; + } + return YES; +} + +// The single colour this layer paints, or NULL. A gradient with distinct stops +// has no single colour, so it reports none rather than inventing one from a stop. +static CGColorRef mob_layer_fill_color(CALayer *layer) { + if ([layer isKindOfClass:[CAShapeLayer class]]) { + CGColorRef fill = ((CAShapeLayer *)layer).fillColor; + if (fill && CGColorGetAlpha(fill) > 0) + return fill; + } + if ([layer isKindOfClass:[CAGradientLayer class]]) { + CAGradientLayer *gradient = (CAGradientLayer *)layer; + if (gradient.colors.count && mob_all_gradient_stops_equal(gradient)) + return (__bridge CGColorRef)gradient.colors.firstObject; + } + if (layer.backgroundColor && CGColorGetAlpha(layer.backgroundColor) > 0) + return layer.backgroundColor; + return NULL; +} + +static CGColorRef mob_layer_text_color(CALayer *layer) { + if ([layer isKindOfClass:[CATextLayer class]]) + return ((CATextLayer *)layer).foregroundColor; + return NULL; +} + +typedef CGColorRef (*MobLayerColorFn)(CALayer *); + +// First colour `probe` finds in this layer subtree, skipping layers owned by +// subviews (they are visited as their own nodes). +static CGColorRef mob_walk_layers(CALayer *layer, NSSet *subviewLayers, MobLayerColorFn probe, + int depth) { + if (!layer || depth > MOB_LAYER_WALK_DEPTH) + return NULL; + CGColorRef own = probe(layer); + if (own) + return own; + for (CALayer *sub in layer.sublayers) { + if ([subviewLayers containsObject:sub]) + continue; + CGColorRef found = mob_walk_layers(sub, subviewLayers, probe, depth + 1); + if (found) + return found; + } + return NULL; +} + +static NSSet *mob_subview_layers(UIView *view) { + NSMutableSet *layers = [NSMutableSet setWithCapacity:view.subviews.count]; + for (UIView *sub in view.subviews) { + if (sub.layer) + [layers addObject:sub.layer]; + } + return layers; +} + +static ERL_NIF_TERM extract_view_bg_color(ErlNifEnv *env, UIView *view) { + if (view.backgroundColor && CGColorGetAlpha(view.backgroundColor.CGColor) > 0) + return argb_term_from_uicolor(env, view.backgroundColor); + CGColorRef painted = + mob_walk_layers(view.layer, mob_subview_layers(view), mob_layer_fill_color, 0); + if (painted) + return argb_term_from_uicolor(env, [UIColor colorWithCGColor:painted]); + return enif_make_atom(env, "nil"); +} + +static ERL_NIF_TERM extract_view_text_color(ErlNifEnv *env, UIView *view) { + if ([view isKindOfClass:[UILabel class]]) + return argb_term_from_uicolor(env, ((UILabel *)view).textColor); + if ([view isKindOfClass:[UITextField class]]) + return argb_term_from_uicolor(env, ((UITextField *)view).textColor); + if ([view isKindOfClass:[UITextView class]]) + return argb_term_from_uicolor(env, ((UITextView *)view).textColor); + if ([view isKindOfClass:[UIButton class]]) + return argb_term_from_uicolor(env, + [(UIButton *)view titleColorForState:UIControlStateNormal]); + CGColorRef painted = + mob_walk_layers(view.layer, mob_subview_layers(view), mob_layer_text_color, 0); + if (painted) + return argb_term_from_uicolor(env, [UIColor colorWithCGColor:painted]); + return enif_make_atom(env, "nil"); +} + static ERL_NIF_TERM build_view_node(ErlNifEnv *env, UIView *view, int depth) { if (!view || depth > 50) return enif_make_atom(env, "nil"); @@ -3758,13 +3921,24 @@ static ERL_NIF_TERM build_view_node(ErlNifEnv *env, UIView *view, int depth) { children = enif_make_list_cell(env, child, children); } - ERL_NIF_TERM keys[5] = {enif_make_atom(env, "type"), enif_make_atom(env, "label"), - enif_make_atom(env, "value"), enif_make_atom(env, "frame"), - enif_make_atom(env, "children")}; - ERL_NIF_TERM vals[5] = {enif_make_atom(env, type_str), nsstring_to_term(env, text), - nsstring_to_term(env, value), frame, children}; + // `class` is the concrete UIView subclass. On SwiftUI it's the only thing + // that says what a node actually is (`type` collapses everything unknown to + // "view"), and it's what tells you which renderer drew a node when a colour + // comes back nil. + ERL_NIF_TERM keys[8] = {enif_make_atom(env, "type"), enif_make_atom(env, "class"), + enif_make_atom(env, "label"), enif_make_atom(env, "value"), + enif_make_atom(env, "frame"), enif_make_atom(env, "bg_color"), + enif_make_atom(env, "text_color"), enif_make_atom(env, "children")}; + ERL_NIF_TERM vals[8] = {enif_make_atom(env, type_str), + nsstring_to_term(env, NSStringFromClass(object_getClass(view))), + nsstring_to_term(env, text), + nsstring_to_term(env, value), + frame, + extract_view_bg_color(env, view), + extract_view_text_color(env, view), + children}; ERL_NIF_TERM result; - enif_make_map_from_arrays(env, keys, vals, 5, &result); + enif_make_map_from_arrays(env, keys, vals, 8, &result); return result; } @@ -3790,20 +3964,164 @@ static ERL_NIF_TERM nif_ui_view_tree(ErlNifEnv *env, int argc, const ERL_NIF_TER // Synthetic root wrapping all top-level windows. Frame is the screen size // so consumers always have a valid bounding box for the whole UI. - ERL_NIF_TERM root_keys[5] = {enif_make_atom(env, "type"), enif_make_atom(env, "label"), - enif_make_atom(env, "value"), enif_make_atom(env, "frame"), - enif_make_atom(env, "children")}; - ERL_NIF_TERM root_vals[5] = { - enif_make_atom(env, "root"), enif_make_atom(env, "nil"), enif_make_atom(env, "nil"), - enif_make_tuple4(env, enif_make_double(env, 0.0), enif_make_double(env, 0.0), - enif_make_double(env, screen_size.width), - enif_make_double(env, screen_size.height)), - windows_list}; + // The synthetic root paints nothing and has no class, so those are always + // nil — but the keys are present so consumers can read them on any node. + ERL_NIF_TERM root_keys[8] = { + enif_make_atom(env, "type"), enif_make_atom(env, "class"), + enif_make_atom(env, "label"), enif_make_atom(env, "value"), + enif_make_atom(env, "frame"), enif_make_atom(env, "bg_color"), + enif_make_atom(env, "text_color"), enif_make_atom(env, "children")}; + ERL_NIF_TERM root_vals[8] = {enif_make_atom(env, "root"), + enif_make_atom(env, "nil"), + enif_make_atom(env, "nil"), + enif_make_atom(env, "nil"), + enif_make_tuple4(env, enif_make_double(env, 0.0), + enif_make_double(env, 0.0), + enif_make_double(env, screen_size.width), + enif_make_double(env, screen_size.height)), + enif_make_atom(env, "nil"), + enif_make_atom(env, "nil"), + windows_list}; ERL_NIF_TERM root; - enif_make_map_from_arrays(env, root_keys, root_vals, 5, &root); + enif_make_map_from_arrays(env, root_keys, root_vals, 8, &root); return root; } +// ── ui_paint_debug/0 — census of where colour lives in this app's view tree ── +// +// When ui_view_tree reports nil colours you need to know *why* before changing +// the extractor: which view classes the renderer produced, what layers hang off +// them, and which colour-bearing properties are actually set. Guessing at +// SwiftUI's private class names across device rebuilds is the slow way to find +// that out; this answers it in one RPC. +// +// Returns a JSON binary, grouped by (view class, layer class, sublayer classes) +// with a count and a tally of which paint properties were non-nil in that group: +// +// {"total_views":443, +// "groups":[{"view":"SwiftUI.CGDrawingView","layer":"SwiftUI.CGDrawingLayer", +// "sublayers":["CAShapeLayer"],"count":40, +// "view_bg":0,"layer_bg":0,"shape_fill":40,"gradient":0, +// "text_layer_fg":0,"uikit_text":0,"has_contents":40}, ...]} +// +// Read it as: for these 40 views, colour is only in CAShapeLayer.fillColor, so +// that is what the extractor has to read. +static void mob_paint_census(UIView *view, NSMutableDictionary *groups, int depth) { + if (!view || depth > 50) + return; + + NSMutableArray *sublayerClasses = [NSMutableArray array]; + NSSet *ownedBySubviews = mob_subview_layers(view); + BOOL shapeFill = NO, gradient = NO, textLayerFg = NO, layerBg = NO, contents = NO; + for (CALayer *sub in view.layer.sublayers) { + if ([ownedBySubviews containsObject:sub]) + continue; + NSString *cls = NSStringFromClass(object_getClass(sub)); + if (![sublayerClasses containsObject:cls]) + [sublayerClasses addObject:cls]; + if ([sub isKindOfClass:[CAShapeLayer class]] && ((CAShapeLayer *)sub).fillColor) + shapeFill = YES; + if ([sub isKindOfClass:[CAGradientLayer class]] && ((CAGradientLayer *)sub).colors.count) + gradient = YES; + if ([sub isKindOfClass:[CATextLayer class]] && ((CATextLayer *)sub).foregroundColor) + textLayerFg = YES; + if (sub.backgroundColor) + layerBg = YES; + if (sub.contents) + contents = YES; + } + if (view.layer.backgroundColor) + layerBg = YES; + if (view.layer.contents) + contents = YES; + if ([view.layer isKindOfClass:[CAShapeLayer class]] && ((CAShapeLayer *)view.layer).fillColor) + shapeFill = YES; + + BOOL uikitText = + [view isKindOfClass:[UILabel class]] || [view isKindOfClass:[UITextField class]] || + [view isKindOfClass:[UITextView class]] || [view isKindOfClass:[UIButton class]]; + + NSString *viewClass = NSStringFromClass(object_getClass(view)); + NSString *layerClass = NSStringFromClass(object_getClass(view.layer)); + NSArray *sortedSublayers = [sublayerClasses sortedArrayUsingSelector:@selector(compare:)]; + NSString *key = [NSString stringWithFormat:@"%@|%@|%@", viewClass, layerClass, + [sortedSublayers componentsJoinedByString:@","]]; + + NSMutableDictionary *group = groups[key]; + if (!group) { + group = [NSMutableDictionary dictionaryWithDictionary:@{ + @"view" : viewClass, + @"layer" : layerClass, + @"sublayers" : sortedSublayers, + @"count" : @0, + @"view_bg" : @0, + @"layer_bg" : @0, + @"shape_fill" : @0, + @"gradient" : @0, + @"text_layer_fg" : @0, + @"uikit_text" : @0, + @"has_contents" : @0 + }]; + groups[key] = group; + } + group[@"count"] = @([group[@"count"] intValue] + 1); + if (view.backgroundColor) + group[@"view_bg"] = @([group[@"view_bg"] intValue] + 1); + if (layerBg) + group[@"layer_bg"] = @([group[@"layer_bg"] intValue] + 1); + if (shapeFill) + group[@"shape_fill"] = @([group[@"shape_fill"] intValue] + 1); + if (gradient) + group[@"gradient"] = @([group[@"gradient"] intValue] + 1); + if (textLayerFg) + group[@"text_layer_fg"] = @([group[@"text_layer_fg"] intValue] + 1); + if (uikitText) + group[@"uikit_text"] = @([group[@"uikit_text"] intValue] + 1); + if (contents) + group[@"has_contents"] = @([group[@"has_contents"] intValue] + 1); + + for (UIView *sub in view.subviews) + mob_paint_census(sub, groups, depth + 1); +} + +static ERL_NIF_TERM nif_ui_paint_debug(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { + (void)argc; + (void)argv; + __block NSData *jsonData = nil; + dispatch_sync(dispatch_get_main_queue(), ^{ + NSMutableDictionary *groups = [NSMutableDictionary dictionary]; + for (UIScene *s in [UIApplication sharedApplication].connectedScenes) { + if (![s isKindOfClass:[UIWindowScene class]]) + continue; + for (UIWindow *w in [(UIWindowScene *)s windows]) { + if (!w.isHidden) + mob_paint_census(w, groups, 0); + } + } + int total = 0; + for (NSDictionary *g in groups.allValues) + total += [g[@"count"] intValue]; + NSArray *sorted = [groups.allValues + sortedArrayUsingComparator:^NSComparisonResult(NSDictionary *a, NSDictionary *b) { + return [b[@"count"] compare:a[@"count"]]; + }]; + jsonData = [NSJSONSerialization dataWithJSONObject:@{ + @"total_views" : @(total), + @"groups" : sorted + } + options:0 + error:nil]; + }); + + if (!jsonData) + return enif_make_tuple2(env, enif_make_atom(env, "error"), + enif_make_atom(env, "encode_failed")); + ErlNifBinary bin; + enif_alloc_binary(jsonData.length, &bin); + memcpy(bin.data, jsonData.bytes, jsonData.length); + return enif_make_binary(env, &bin); +} + // ── screen_info/0 — unified screen/safe-area shape ─────────────────────────── // // Returns: %{width, height, scale, safe_area: %{top, bottom, left, right}} @@ -4691,6 +5009,26 @@ static ERL_NIF_TERM nif_tap_xy_probe(ErlNifEnv *env) { return list; } +// Block until a UI event reaches the BEAM, or timeout_ms elapses. +// +// Synthetic input is delivered on the main runloop; the SwiftUI gesture handler +// that ends up calling mob_send_tap has usually NOT run by the time the +// dispatch_sync that injected the touch returns. Polling (rather than a fixed +// sleep) keeps a landed tap fast — the common case returns in one step. +static BOOL mob_await_ui_event(uint64_t seq_before, int timeout_ms) { + for (int waited = 0; waited < timeout_ms; waited += 5) { + if (mob_ui_event_seq() != seq_before) + return YES; + [NSThread sleepForTimeInterval:0.005]; + } + return mob_ui_event_seq() != seq_before; +} + +// How long tap_xy waits for the app to react before reporting :no_effect. +// 300ms is well past a SwiftUI tap gesture's recognition delay while staying +// short enough for the tight loops the harness runs. +#define MOB_TAP_SETTLE_MS 300 + static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { // Diagnostics mode — pass :probe or :enumerate_touch or :enumerate_event if (enif_is_atom(env, argv[0])) { @@ -4797,6 +5135,35 @@ static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv CGPoint pt = CGPointMake(x, y); + // Sampled before any injection so we can tell a tap that ran a handler from + // one the input API merely accepted. See mob_note_ui_event. + const uint64_t seq_before = mob_ui_event_seq(); + + // Hit-test on both platforms first: a coordinate outside every visible window + // can never do anything, and saying so beats reporting :no_effect. + __block UIWindow *targetWindow = nil; + __block UIView *hitView = nil; + dispatch_sync(dispatch_get_main_queue(), ^{ + for (UIScene *scene in [UIApplication sharedApplication].connectedScenes) { + if (![scene isKindOfClass:[UIWindowScene class]]) + continue; + for (UIWindow *win in [(UIWindowScene *)scene windows]) { + if (win.isHidden) + continue; + UIView *hit = [win hitTest:pt withEvent:nil]; + if (hit) { + targetWindow = win; + hitView = hit; + return; + } + } + } + }); + if (!hitView) { + return enif_make_tuple2(env, enif_make_atom(env, "error"), + enif_make_atom(env, "no_view_at_point")); + } + #if TARGET_OS_SIMULATOR // ── Simulator: accessibility-based activation by coordinates ───────────────── // The iOS simulator rejects in-process synthetic IOHIDEvents (no valid display @@ -4804,6 +5171,12 @@ static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv // proper event system backing. Accessibility activation is the reliable path // for the simulator; for scroll views and custom GRs that lack accessibility, // a simulator-specific event injection mechanism would be needed. + // + // accessibilityActivate returning YES does NOT mean the app reacted: SwiftUI + // only maps it to a default action for Button-like views. A plain + // `.onTapGesture` (what Mob's Box/Row/Column use for on_tap) has no AX action, + // so activation "succeeds" and the handler never fires. That is why the + // outcome is decided by the event counter below, not by `activated`. __block BOOL activated = NO; dispatch_sync(dispatch_get_main_queue(), ^{ for (UIScene *scene in [UIApplication sharedApplication].connectedScenes) { @@ -4814,16 +5187,16 @@ static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv continue; id elem = find_a11y_at_point(win, pt, 0); if (elem) { - LOGI(@"tap_xy(sim): accessibilityActivate on %@ frame=%@", + LOGI(@"tap_xy(sim): accessibilityActivate on %@ frame=%@ window=%@", NSStringFromClass(object_getClass(elem)), - NSStringFromCGRect([elem accessibilityFrame])); + NSStringFromCGRect([elem accessibilityFrame]), + NSStringFromClass(object_getClass(targetWindow))); [elem accessibilityActivate]; // For text fields: accessibilityActivate on UITextFieldLabel // (the hint label inside UITextField) doesn't focus the // field. Walk the responder chain up from the hit view to // find the first UITextField/UITextView and focus it. - UIView *hv = [win hitTest:pt withEvent:nil]; - UIResponder *r = hv; + UIResponder *r = hitView; while (r) { if ([r isKindOfClass:[UITextField class]] || [r isKindOfClass:[UITextView class]]) { @@ -4838,38 +5211,18 @@ static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv } } }); - if (activated) - return enif_make_atom(env, "ok"); - return enif_make_tuple2(env, enif_make_atom(env, "error"), - enif_make_atom(env, "no_element_at_point")); - -#else - // ── Real device: UITouch injection via IOHIDEvent ───────────────────────────── - __block UIWindow *targetWindow = nil; - __block UIView *hitView = nil; - - dispatch_sync(dispatch_get_main_queue(), ^{ - for (UIScene *scene in [UIApplication sharedApplication].connectedScenes) { - if (![scene isKindOfClass:[UIWindowScene class]]) - continue; - for (UIWindow *win in [(UIWindowScene *)scene windows]) { - if (win.isHidden) - continue; - UIView *hit = [win hitTest:pt withEvent:nil]; - if (hit) { - targetWindow = win; - hitView = hit; - return; - } - } - } - }); - - if (!hitView) { + if (!activated) { return enif_make_tuple2(env, enif_make_atom(env, "error"), - enif_make_atom(env, "no_view_at_point")); + enif_make_atom(env, "no_element_at_point")); + } + if (!mob_await_ui_event(seq_before, MOB_TAP_SETTLE_MS)) { + return enif_make_tuple2(env, enif_make_atom(env, "error"), + enif_make_atom(env, "no_effect")); } + return enif_make_atom(env, "ok"); +#else + // ── Real device: UITouch injection via IOHIDEvent ───────────────────────────── __block BOOL ok = NO; dispatch_sync(dispatch_get_main_queue(), ^{ ok = mob_send_touch_phase(targetWindow, hitView, pt, UITouchPhaseBegan); @@ -4884,6 +5237,15 @@ static ERL_NIF_TERM nif_tap_xy(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv mob_send_touch_phase(targetWindow, hitView, pt, UITouchPhaseEnded); }); + // mob_send_touch_phase's YES only means "the private input API exists and + // accepted the event" — on iOS 26 devices UIKit routinely swallows the + // in-process IOHID event and no touch is ever delivered (decisions/ + // 2026-08-09-ios-device-tap-injection-has-no-effect.md). The counter is the + // only thing that distinguishes a real tap from an accepted no-op. + if (!mob_await_ui_event(seq_before, MOB_TAP_SETTLE_MS)) { + return enif_make_tuple2(env, enif_make_atom(env, "error"), + enif_make_atom(env, "no_effect")); + } return enif_make_atom(env, "ok"); #endif } @@ -5418,6 +5780,44 @@ static void mob_collect_scroll_views(UIView *view, NSMutableArray {ok, PixelW, PixelH, RGBA} | {error, Reason} +// +// Raw pixels for one region of the app's own surface. This is the only reliable +// way to verify what colour was actually drawn: on iOS 26 SwiftUI paints through +// SDFLayer or rasterises into `contents`, so neither the view nor the layer tree +// exposes a readable colour — see +// decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md. +// +// The crop happens in the render, not after it, so what crosses distribution is +// one element's worth of pixels instead of a whole framebuffer. Coordinates are +// window points (the same space element_frames/0 reports); the returned buffer is +// PixelW*PixelH*4 bytes of 8-bit RGBA at the native screen scale, so its size is +// W*H*scale^2*4 — a 100x50pt element is ~180 KB at 3x. +// +// Rects are clamped to the window, so a partly-scrolled-off element samples the +// visible part and reports the pixel dimensions it actually got. A rect entirely +// outside the window is `offscreen` rather than a plausible-looking black. +// +// Unlike screenshot/3 this stays strictly debug-only: it is a screen-capture +// primitive, and a release build shipping it could reconstruct the screen region +// by region, silently defeating the MOB_ENABLE_SCREENSHOT opt-in. +static ERL_NIF_TERM nif_sample_region(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { + (void)argc; + double x = 0.0, y = 0.0, w = 0.0, h = 0.0; + if (!enif_get_double(env, argv[0], &x) || !enif_get_double(env, argv[1], &y) || + !enif_get_double(env, argv[2], &w) || !enif_get_double(env, argv[3], &h)) + return enif_make_badarg(env); + + __block const char *err = NULL; + __block NSMutableData *rgba = nil; + __block size_t pw = 0, ph = 0; + + if (w <= 0.0 || h <= 0.0) + err = "empty_region"; + + if (!err) + dispatch_sync(dispatch_get_main_queue(), ^{ + UIWindow *window = mob_capture_window(); + if (!window) { + err = "no_window"; + return; + } + CGRect crop = CGRectIntersection(window.bounds, CGRectMake(x, y, w, h)); + if (CGRectIsNull(crop) || CGRectIsEmpty(crop)) { + err = "offscreen"; + return; + } + + CGImageRef cg = mob_capture_image(window, crop, 1.0).CGImage; + if (!cg) { + err = "capture_failed"; + return; + } + pw = CGImageGetWidth(cg); + ph = CGImageGetHeight(cg); + + // Redraw into a bitmap context of known layout: a UIImage's own backing + // store has no guaranteed byte order or component count, so reading it + // directly would make the returned bytes device-dependent. + size_t stride = pw * 4; + rgba = [NSMutableData dataWithLength:stride * ph]; + CGColorSpaceRef cs = CGColorSpaceCreateDeviceRGB(); + CGContextRef bmp = + CGBitmapContextCreate(rgba.mutableBytes, pw, ph, 8, stride, cs, + kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big); + CGColorSpaceRelease(cs); + if (!bmp) { + rgba = nil; + err = "capture_failed"; + return; + } + CGContextDrawImage(bmp, CGRectMake(0, 0, pw, ph), cg); + CGContextRelease(bmp); + }); + + if (err || !rgba) + return enif_make_tuple2(env, enif_make_atom(env, "error"), + enif_make_atom(env, err ?: "capture_failed")); + + ErlNifBinary bin; + enif_alloc_binary(rgba.length, &bin); + memcpy(bin.data, rgba.mutableBytes, rgba.length); + return enif_make_tuple4(env, enif_make_atom(env, "ok"), enif_make_ulong(env, pw), + enif_make_ulong(env, ph), enif_make_binary(env, &bin)); +} + static ERL_NIF_TERM nif_scroll_info(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) { ErlNifBinary idb; if (!enif_inspect_binary(env, argv[0], &idb)) @@ -6268,11 +6731,15 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF // * ui_tree — recursive UIAccessibility walk (variable, can be 10s of ms) // * ui_debug — same walk, more output // -// Synthetic-input NIFs (tap_xy, swipe_xy, long_press_xy, type_text, key_press, +// Synthetic-input NIFs (swipe_xy, long_press_xy, type_text, key_press, // delete_backward, clear_text) dispatch_sync to the main queue but also do // some pre-dispatch work; they're left on regular schedulers for now because // the test harness calls them in tight loops and dirty-dispatch overhead would // add up. Re-evaluate if benchmarks show scheduler stalls under heavy harness use. +// +// tap_xy is the exception: it blocks up to MOB_TAP_SETTLE_MS waiting for the +// app to react (that wait is what makes its :ok trustworthy), which is far too +// long to hold a normal scheduler. static ErlNifFunc nif_funcs[] = { #if !MOB_RELEASE // ── Test harness (listed first to survive linker dead-code stripping) ────── @@ -6282,12 +6749,13 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF // App Store validator rejects binaries that reference them). {"ui_tree", 0, nif_ui_tree, ERL_NIF_DIRTY_JOB_CPU_BOUND}, {"ui_view_tree", 0, nif_ui_view_tree, ERL_NIF_DIRTY_JOB_CPU_BOUND}, + {"ui_paint_debug", 0, nif_ui_paint_debug, ERL_NIF_DIRTY_JOB_CPU_BOUND}, {"ui_debug", 0, nif_ui_debug, ERL_NIF_DIRTY_JOB_CPU_BOUND}, {"screen_info", 0, nif_screen_info, 0}, {"tap", 1, nif_tap, 0}, {"ax_action", 2, nif_ax_action, 0}, {"ax_action_at_xy", 3, nif_ax_action_at_xy, 0}, - {"tap_xy", 2, nif_tap_xy, 0}, + {"tap_xy", 2, nif_tap_xy, ERL_NIF_DIRTY_JOB_IO_BOUND}, {"type_text", 1, nif_type_text, 0}, {"delete_backward", 0, nif_delete_backward, 0}, {"key_press", 1, nif_key_press, 0}, @@ -6299,6 +6767,7 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF {"screenshot", 3, nif_screenshot, ERL_NIF_DIRTY_JOB_CPU_BOUND}, #endif #if !MOB_RELEASE + {"sample_region", 4, nif_sample_region, ERL_NIF_DIRTY_JOB_CPU_BOUND}, {"scroll_info", 1, nif_scroll_info, 0}, {"scroll_to", 3, nif_scroll_to, 0}, {"element_frames", 0, nif_element_frames, ERL_NIF_DIRTY_JOB_CPU_BOUND}, diff --git a/lib/mob/test.ex b/lib/mob/test.ex index 5b20157..a7a9f5e 100644 --- a/lib/mob/test.ex +++ b/lib/mob/test.ex @@ -39,6 +39,9 @@ defmodule Mob.Test do Mob.Test.frame(node, "save") # {x, y, w, h} Mob.Test.tap_id(node, "save") # drive by id at real coords + # What colour did the app actually draw? (samples pixels — the view tree can't) + Mob.Test.sample_color(node, "my-card") # %{average: 0xFF2196F3, ...} + # Device API simulation Mob.Test.send_message(node, {:permission, :camera, :granted}) Mob.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}}) @@ -73,7 +76,7 @@ defmodule Mob.Test do | API | Source | When to use | |-------------------------------|-------------------------------------|-------------| | `tree/1`, `find/2` | Mob render tree (logical components) | Mob apps you control. Fast, exact, has `on_tap` tags, no AX activation needed. | - | `view_tree/1`, `find_view/2` | Native view hierarchy via NIF | Native pixel frames; works for any app on iOS UIKit; shallow on SwiftUI/Compose. | + | `view_tree/1`, `find_view/2` | Native view hierarchy via NIF | Native pixel frames **and painted colours**; works for any app on iOS UIKit; shallow on SwiftUI/Compose. | | `ui_tree/1` | OS accessibility tree | What sighted users read; works on any app *if* AX is active (iOS: VoiceOver). Strict superset of `view_tree` for UIKit; the only path to semantics inside SwiftUI/Compose. | Choose render tree first if your app is Mob-rendered. Reach for `view_tree` @@ -111,29 +114,48 @@ defmodule Mob.Test do | `back/1`, `pop/1`, `navigate`| ✅ | ✅ | ✅ | | `send_message/2` | ✅ | ✅ | ✅ | | `screen_info/1` | ✅ | ✅ | ✅ | - | `view_tree/1` | ✅ (shallow†) | ✅ (shallow†) | ✅ (root only‡) | - | `find_view/2` | ✅ | ✅ | ✅ | + | `view_tree/1` | ✅ (shallow†) | ✅ (shallow†) | ❌ not_loaded‡ | + | `sample_color/2` | ✅ | ✅ | ❌ not_loaded° | + | `find_view/2` | ✅ | ✅ | ❌ not_loaded‡ | | `ui_tree/1` (legacy AX) | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_loaded | | `ax_action/3` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported | | `ax_action_at_xy/4` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported | | `toggle/2` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable | | `dismiss_alert/2` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable | | `adjust_slider/4` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable | - | `tap_xy/3` | ✅ (AX path) | ✅ (HID inj.) | n/a | - | `swipe/5` | ⚠️ scroll only| ✅ (HID inj.) | n/a | + | `tap_xy/3` | ⚠️ AX-activatable only¶ | ❌ no_effect¶ | n/a | + | `swipe/5` | ⚠️ scroll only| ⚠️ unverified✱| n/a | - **†** SwiftUI doesn't expose its content as separate UIView instances — `view_tree` reaches the SwiftUI hosting view's container and stops. For semantic content on Mob screens use `tree/1` (render tree); for any other SwiftUI-based content use `ui_tree/1`. - - **‡** Android's Mob renderer is Compose. The View walk stops at the - `AndroidComposeView` host. The eventual fix is `Modifier.onGloballyPositioned` - in Mob's components writing to a registry the NIF reads (planned). - See `issues.md` #11. + - **‡** Android's `ui_view_tree` NIF delegates to a `MobBridge.uiViewTree()` + Kotlin method that no shipped template implements, so it returns + `{:error, :not_loaded}`. When it lands it must emit the same keys iOS does + (including `bg_color`/`text_color`); the contract is documented at the NIF + in `android/jni/mob_nif.zig`. The Mob renderer is Compose, so the View walk + would stop at the `AndroidComposeView` host anyway — the real fix is + `Modifier.onGloballyPositioned` in Mob's components writing to a registry + the NIF reads. See `issues.md` #11. + - **°** `sample_region/4` is implemented in `ios/mob_nif.m` only. Android + would need the same crop-in-the-render treatment against the activity + window; until then `sample_color/2` returns `{:error, {:badrpc, _}}` there. - **§** "AX active" means an iOS accessibility client is asking for the AX tree so SwiftUI materializes it. Today: VoiceOver toggle. Production: `XCAXClient_iOS` activation, debug-only — see WireTap stretch goals in `future_developments.md`. + - **¶** `tap_xy/3` now verifies that the tap actually produced an event + before returning `:ok`. On the simulator that limits it to elements SwiftUI + exposes an accessibility action for (`Button`, text fields) — a `Box` with + `on_tap:` returns `{:error, :no_effect}`. On a physical device the + IOHID-injected touch is accepted but never delivered, so **every** + coordinate returns `{:error, :no_effect}`. Drive taps with `tap/2` (by tag); + see `tap_xy/3` and + `decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md`. + - **✱** `swipe/5` and `long_press/4` use the same device injection path as + `tap_xy/3` and still report `:ok` on acceptance rather than on effect. + Same root cause, not yet converted — treat their `:ok` as unverified. Helpers that depend on AX return clear error tuples on Android instead of raising. Callers should match on `{:error, :not_supported_on_android}` and @@ -381,50 +403,179 @@ defmodule Mob.Test do Returns a nested map: %{ - type: :root, label: nil, value: nil, + type: :root, class: nil, label: nil, value: nil, frame: {0.0, 0.0, 393.0, 852.0}, + bg_color: nil, text_color: nil, children: [ - %{type: :window, ..., children: [ + %{type: :window, class: "UIWindow", ..., children: [ %{type: :scroll, ..., children: [ - %{type: :button, label: "Roll Dice", - frame: {24.0, 416.0, 327.0, 53.5}, children: []} + %{type: :button, class: "SwiftUI.CGDrawingView", label: "Roll Dice", + frame: {24.0, 416.0, 327.0, 53.5}, + bg_color: 0xFF2196F3, text_color: 0xFFFFFFFF, children: []} ]} ]} ] } - On Android, the JSON returned by `mob_nif:ui_view_tree/0` is decoded here. + `:class` is the concrete native view class. On SwiftUI it is usually the only + thing that identifies a node — `:type` collapses anything it doesn't recognise + to `:view` — and it's what tells you which renderer drew a node when a colour + comes back `nil`. + + ## Colours + + `:bg_color` and `:text_color` are the colours the view **actually painted**, + as `0xAARRGGBB` integers — the same representation component props use + (`guides/theming.md`). `nil` means nothing paintable was found, or the colour + has no single RGBA value (a multi-stop gradient, a pattern fill). + + UIKit puts colour on the view (`UIView.backgroundColor`, `UILabel.textColor`). + **SwiftUI mostly does not** — `.background(Color, in: shape)` and + `.foregroundColor`, which is what Mob's renderer uses for every Box and Text, + go through SwiftUI's own renderer and land on a `CALayer` (typically a + `CAShapeLayer` fill) under a structural view whose own `backgroundColor` stays + `nil`. So each node also harvests from its own layer subtree, excluding layers + owned by its subviews so a container never claims a child's paint. + + Sources consulted per node, first match wins: + + | | Background | Text | + |---|---|---| + | view | `UIView.backgroundColor` | `UILabel`/`UITextField`/`UITextView`/`UIButton` | + | layer subtree | `CAShapeLayer.fillColor`, single-stop `CAGradientLayer`, `CALayer.backgroundColor` | `CATextLayer.foregroundColor` | + + Fully-transparent colours are treated as no colour, so a `Color.clear` + placeholder doesn't read as "painted black at alpha 0". + + Because these are read back off `UIView`/`CALayer` rather than echoed from the + render tree, they are the way to catch a styling regression where a theme or + modifier silently drops a colour Elixir sent. Compare against `tree/1` (what + Elixir asked for) to see the two diverge. + + **If colours come back `nil` across the board**, don't guess at the reason — + call `paint_debug/1`, which reports which view/layer classes the renderer + produced and which colour properties they actually set. On iOS 26 SwiftUI that + is the expected outcome, and `sample_color/2` (real pixels) is the way to + verify a drawn colour. + + On Android, the JSON returned by `mob_nif:ui_view_tree/0` is decoded here — + but no shipped `MobBridge.kt` implements `uiViewTree()`, so today Android + returns `{:error, :not_loaded}`. """ @spec view_tree(node()) :: map() | {:error, term()} def view_tree(node) do case :rpc.call(node, :mob_nif, :ui_view_tree, []) do - bin when is_binary(bin) -> :json.decode(bin) |> normalize_tree() + bin when is_binary(bin) -> bin |> :json.decode() |> normalize_view_tree() %{} = m -> m other -> other end end - # JSON decode produces string keys; the iOS NIF returns atom keys directly. - # Normalize to atom keys so the API is uniform across platforms. - defp normalize_tree(%{"type" => _} = node) do + @doc """ + Normalize an Android-shaped (JSON-decoded, string-keyed) view tree into the + iOS map shape: atom keys, atom `:type`, `{x, y, w, h}` frame tuple. + + `view_tree/1` applies this automatically. It's public so a captured tree can + be normalized without a device. + """ + @spec normalize_view_tree(map() | term()) :: map() | term() + def normalize_view_tree(%{"type" => _} = node) do %{ type: normalize_atom(node["type"]), - label: node["label"], - value: node["value"], + class: denull(node["class"]), + label: denull(node["label"]), + value: denull(node["value"]), frame: case node["frame"] do [x, y, w, h] -> {x * 1.0, y * 1.0, w * 1.0, h * 1.0} - other -> other + other -> denull(other) end, - children: Enum.map(node["children"] || [], &normalize_tree/1) + bg_color: denull(node["bg_color"]), + text_color: denull(node["text_color"]), + children: Enum.map(denull(node["children"]) || [], &normalize_view_tree/1) } end - defp normalize_tree(other), do: other + def normalize_view_tree(other), do: other + + # `:json.decode/1` maps JSON null to the atom :null. Left as-is it leaks into + # every comparison against nil, and `:null || []` is truthy, so an absent + # children list would crash Enum.map. + defp denull(:null), do: nil + defp denull(other), do: other defp normalize_atom(s) when is_binary(s), do: String.to_atom(s) defp normalize_atom(a) when is_atom(a), do: a + @doc """ + Census of where colour lives in the native view tree — the diagnostic to reach + for when `view_tree/1` reports `nil` colours and you need to know why. + + Groups every native view by `(view class, layer class, sublayer classes)` and + reports, per group, how many views set each colour-bearing property: + + Mob.Test.paint_debug(node) + #=> %{ + # "total_views" => 443, + # "groups" => [ + # %{"view" => "SwiftUI.CGDrawingView", "layer" => "SwiftUI.CGDrawingLayer", + # "sublayers" => ["CAShapeLayer"], "count" => 40, + # "view_bg" => 0, "layer_bg" => 0, "shape_fill" => 40, + # "gradient" => 0, "text_layer_fg" => 0, "uikit_text" => 0, + # "has_contents" => 40}, + # ... + # ] + # } + + Read a row as: for these 40 views the only colour set is + `CAShapeLayer.fillColor`, so that is the property the extractor has to read. + A group where every tally is 0 but `has_contents` is high is a view that drew + itself into a bitmap — its colour is not recoverable without pixel sampling. + + iOS only, debug builds only. Android raises `:nif_error`. + """ + @spec paint_debug(node()) :: map() | {:error, term()} + def paint_debug(node) do + case :rpc.call(node, :mob_nif, :ui_paint_debug, []) do + bin when is_binary(bin) -> :json.decode(bin) + other -> other + end + end + + @doc """ + Tally of the distinct painted colours in a view tree — the cheap way to assert + a styling change actually reached the screen. + + Pass a node to fetch the tree, or an already-fetched tree to work offline. + Returns `%{background: %{argb => count}, text: %{argb => count}}`, `nil` + colours excluded. + + Mob.Test.color_census(node) + #=> %{background: %{0xFF2196F3 => 4, 0xFF1E1E1E => 1}, text: %{0xFFFFFFFF => 9}} + + A theme regression that discards backgrounds shows up as an empty (or + collapsed) `:background` map, and two themes that should differ produce + different key sets. + """ + @spec color_census(node() | map()) :: %{background: map(), text: map()} + def color_census(node) when is_atom(node), do: color_census(view_tree(node)) + + def color_census(%{} = tree) do + tree + |> flatten_tree() + |> Enum.reduce(%{background: %{}, text: %{}}, fn {_path, n}, acc -> + acc + |> tally(:background, n[:bg_color]) + |> tally(:text, n[:text_color]) + end) + end + + defp tally(acc, _key, nil), do: acc + + defp tally(acc, key, color) do + Map.update!(acc, key, &Map.update(&1, color, 1, fn n -> n + 1 end)) + end + @doc """ Return the view tree flattened to a list of `{path, node}` tuples. @@ -717,12 +868,48 @@ defmodule Mob.Test do end @doc """ - Tap at screen coordinates on the native app. On simulator uses accessibility - activation; on real device synthesises a UITouch via IOHIDEvent. + Tap at screen coordinates on the native app. Mob.Test.tap_xy(node, 289.7, 518.8) + + ## Return values + + `:ok` means **the app reacted** — an event reached the BEAM within 300ms of + the tap. Every other outcome is an error tuple; there is no "probably worked". + + | Value | Meaning | + |---|---| + | `:ok` | A `tap`/`focus`/`change`/`submit`/`select` event reached the BEAM. | + | `{:error, :no_view_at_point}` | Hit-test found nothing — the coordinate is outside every visible window. | + | `{:error, :no_element_at_point}` | iOS simulator only: a view is there but no accessibility element to activate. | + | `{:error, :no_effect}` | Input was accepted by the OS but no handler ran. | + | `{:error, probe}` | iOS device only: the private injection API is missing; `probe` lists which selectors resolved. | + + ## Real capability per platform — read before trusting a result + + - **iOS simulator** — activates the accessibility element under the point. + That works for `Button`, and for text fields (the responder chain is walked + to focus them). It does **not** work for Mob's `Box`/`Row`/`Column` with + `on_tap:`: SwiftUI gives a plain `.onTapGesture` no accessibility action, so + activation is accepted and the handler never runs. Those taps return + `{:error, :no_effect}`. Use `tap/2` (by tag) to drive them. + - **iOS physical device** — synthesises an `IOHIDEvent`. As of iOS 26.5 UIKit + accepts the event and delivers no touch, so this returns + `{:error, :no_effect}` for every coordinate. Treat coordinate tapping as + **not working on device** and use `tap/2`. See + `decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md`. + - **Android** — not routed through this function; `adb shell input tap` works + and is what the tooling uses. + + ## `:no_effect` in sidecar mode + + The check is "did an event reach the BEAM", so it only sees handlers Mob owns. + Driving a non-Mob app (sidecar mode), a genuinely successful tap still reports + `{:error, :no_effect}` because there is nothing for the NIF to observe. + Confirm those with `ui_tree/1` or a screenshot instead. """ - @spec tap_xy(node(), number(), number()) :: :ok | {:error, atom()} + @spec tap_xy(node(), number(), number()) :: + :ok | {:error, :no_view_at_point | :no_element_at_point | :no_effect | term()} def tap_xy(node, x, y) do :rpc.call(node, :mob_nif, :tap_xy, [x * 1.0, y * 1.0]) end @@ -1150,6 +1337,9 @@ defmodule Mob.Test do (see `element_frames/1`). Mob.Test.tap_id(node, "save") + + Inherits `tap_xy/3`'s return contract, including its platform limits — read + those before treating a non-`:ok` result as a test failure. """ @spec tap_id(node(), String.t() | atom()) :: :ok | {:error, term()} def tap_id(node, id) do @@ -1160,6 +1350,152 @@ defmodule Mob.Test do end end + # ── Colour sampling (pixels, because the view tree can't answer) ───────────── + + @doc """ + What colour did the app actually draw in a region? Samples real pixels. + + Address the region either by an element `:id` (resolved through + `element_frames/1`) or by an explicit `{x, y, w, h}` rect in logical points: + + Mob.Test.sample_color(node, "my-card") + Mob.Test.sample_color(node, {24.0, 416.0, 327.0, 53.5}) + + Returns `{:ok, sample}` where `sample` is the map `reduce_rgba/3` produces: + + {:ok, %{average: 0xFF2196F3, dominant: 0xFF2196F3, dominant_share: 0.94, + distinct: 37, pixels: 2400}} + + ## Why pixels and not `view_tree/1` + + `view_tree/1`'s `:bg_color` is `nil` for virtually all SwiftUI content — on + iOS 26 SwiftUI paints via `SDFLayer` or rasterises into `contents`, exposing no + readable paint property (measured: colour for 4 of 443 nodes; see + `decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md`). Sampling + the rendered pixels is the only way to catch a regression like the glass theme + that discarded every Box background — under which a `background: :primary` Box + and a `:surface_raised` Box sample to the *same* colour, and that difference is + what this asserts. + + ## Reading the result + + A region is rarely one flat colour — a card has text, a border, antialiased + corners — so a bare mean can be misleading. `:average` is the mean, `:dominant` + is the most common exact pixel value (the background of a mostly-flat region), + and `:dominant_share` says how much to trust it: `0.9` is a flat fill, `0.2` is + a gradient or a busy region where only `:average` means much. Assert on + `:dominant` for solid fills, on `:average` for anything glassy. + + ## Errors + + * `{:error, :not_found}` — no element with that `:id` has a tracked frame + (the element needs an `:id`, and must have laid out at least once) + * `{:error, :empty_frame}` — the element's frame has zero width or height + * `{:error, :offscreen}` — the rect lies entirely outside the window + * `{:error, :no_window}` — app has no visible window (backgrounded) + * `{:error, :size_mismatch}` — the buffer didn't match the reported + dimensions, so no colour is reported rather than a wrong one + * `{:error, {:badrpc, _}}` — no `sample_region/4` on this platform; the NIF + is iOS-only and debug-build only + + A rect that only partly overlaps the window is clamped to the visible part and + `:pixels` reports what was actually sampled. + + The payload is `w * h * screen_scale^2 * 4` bytes — cropping happens in the + native render, so an element-sized region is tens to hundreds of KB, not a + framebuffer. Don't hand it a full-screen rect. + """ + @spec sample_color(node(), String.t() | atom() | {number(), number(), number(), number()}) :: + {:ok, map()} | {:error, term()} + def sample_color(node, id_or_rect) + + def sample_color(node, {x, y, w, h}) + when is_number(x) and is_number(y) and is_number(w) and is_number(h) do + sample_rect(node, x, y, w, h) + end + + def sample_color(node, id) do + case frame(node, id) do + {_x, _y, w, h} when w <= 0.0 or h <= 0.0 -> {:error, :empty_frame} + {x, y, w, h} -> sample_rect(node, x, y, w, h) + nil -> {:error, :not_found} + {:error, _} = err -> err + end + end + + defp sample_rect(node, x, y, w, h) do + args = [x * 1.0, y * 1.0, w * 1.0, h * 1.0] + + case :rpc.call(node, :mob_nif, :sample_region, args) do + {:ok, pixel_w, pixel_h, rgba} -> reduce_rgba(rgba, pixel_w, pixel_h) + {:error, _} = err -> err + other -> {:error, other} + end + end + + @doc """ + Reduce a raw RGBA buffer to colour statistics. Pure — no device needed. + + `rgba` is `width * height` pixels of 4 bytes each in R, G, B, A order (what + `:mob_nif.sample_region/4` returns). Colours come back as `0xAARRGGBB` + integers, alpha first, matching component props (`guides/theming.md`). + + Mob.Test.reduce_rgba(<<0, 0, 255, 255, 0, 0, 255, 255>>, 2, 1) + #=> {:ok, %{average: 0xFF0000FF, dominant: 0xFF0000FF, dominant_share: 1.0, + # distinct: 1, pixels: 2}} + + `:average` is the per-channel mean (each channel independently, alpha + included, rounded to nearest). `:dominant` is the most frequent exact pixel + value, ties broken by the higher `0xAARRGGBB` value so the result is + deterministic. `:dominant_share` is its fraction of all pixels and + `:distinct` counts distinct values — together they say whether `:dominant` + describes a flat fill or just the most common pixel of a gradient. + + The capture path renders opaque, so alpha is `255` in practice; a buffer with + varying alpha is averaged channel-wise and *not* un-premultiplied. + + `{:error, :empty_region}` for a non-positive dimension, `{:error, + :size_mismatch}` when `byte_size(rgba) != width * height * 4`. + """ + @spec reduce_rgba(binary(), integer(), integer()) :: {:ok, map()} | {:error, atom()} + def reduce_rgba(rgba, width, height) + when is_binary(rgba) and is_integer(width) and is_integer(height) do + pixels = width * height + + cond do + width <= 0 or height <= 0 -> {:error, :empty_region} + byte_size(rgba) != pixels * 4 -> {:error, :size_mismatch} + true -> {:ok, rgba_stats(rgba, pixels)} + end + end + + defp rgba_stats(rgba, pixels) do + {sum_a, sum_r, sum_g, sum_b, freq} = + for <>, reduce: {0, 0, 0, 0, %{}} do + {sum_a, sum_r, sum_g, sum_b, freq} -> + {sum_a + a, sum_r + r, sum_g + g, sum_b + b, + Map.update(freq, argb(a, r, g, b), 1, &(&1 + 1))} + end + + {dominant, count} = Enum.max_by(freq, fn {color, n} -> {n, color} end) + + %{ + average: + argb( + round(sum_a / pixels), + round(sum_r / pixels), + round(sum_g / pixels), + round(sum_b / pixels) + ), + dominant: dominant, + dominant_share: count / pixels, + distinct: map_size(freq), + pixels: pixels + } + end + + defp argb(a, r, g, b), do: a * 0x1000000 + r * 0x10000 + g * 0x100 + b + # ── Native UI (requires MCP tools) ─────────────────────────────────────────── @doc """ diff --git a/src/mob_nif.erl b/src/mob_nif.erl index c7fed54..d6adaca 100644 --- a/src/mob_nif.erl +++ b/src/mob_nif.erl @@ -85,6 +85,7 @@ %% Test harness — native UI inspection and interaction ui_tree/0, ui_view_tree/0, + ui_paint_debug/0, ui_debug/0, screen_info/0, tap/1, @@ -101,6 +102,7 @@ %% (remote-driving: agent gets pixels + deterministic scroll over dist, %% no adb/xcrun). See Mob.Test.screenshot/2, scroll_info/2, scroll_to/3. screenshot/3, + sample_region/4, scroll_info/1, scroll_to/3, element_frames/0, @@ -185,6 +187,7 @@ device_keep_awake/1, ui_tree/0, ui_view_tree/0, + ui_paint_debug/0, ui_debug/0, screen_info/0, tap/1, @@ -198,6 +201,7 @@ long_press_xy/3, swipe_xy/4, screenshot/3, + sample_region/4, scroll_info/1, scroll_to/3, element_frames/0, @@ -310,12 +314,26 @@ device_orientation() -> erlang:nif_error(not_loaded). device_lock_orientation(_Orientation) -> erlang:nif_error(not_loaded). device_keep_awake(_On) -> erlang:nif_error(not_loaded). ui_tree() -> erlang:nif_error(not_loaded). +%% ui_view_tree() -> nested map (iOS) | JSON binary (Android) | {error, Reason} +%% Node shape: #{type, label, value, frame, bg_color, text_color, children}. +%% bg_color/text_color are the colours the view actually painted, as +%% 0xAARRGGBB integers (see guides/theming.md), or nil. ui_view_tree() -> erlang:nif_error(not_loaded). +%% ui_paint_debug() -> JSON binary censusing where colour lives in the native +%% view tree, grouped by view/layer class. Diagnostic for when ui_view_tree +%% reports nil colours — tells you which property the renderer actually set. +%% iOS only; Android has no implementation, so the stub raises there. +ui_paint_debug() -> erlang:nif_error(not_loaded). ui_debug() -> erlang:nif_error(not_loaded). screen_info() -> erlang:nif_error(not_loaded). tap(_Label) -> erlang:nif_error(not_loaded). ax_action(_Match, _Action) -> erlang:nif_error(not_loaded). ax_action_at_xy(_X, _Y, _Action) -> erlang:nif_error(not_loaded). +%% tap_xy(X, Y) -> ok | {error, Reason} +%% ok is only returned when the tap demonstrably reached the BEAM (a tap/focus/ +%% change/submit/select event arrived within 300ms). Reason is one of +%% no_view_at_point | no_element_at_point (simulator) | no_effect | Probe. +%% See Mob.Test.tap_xy/3 for the per-platform capabilities behind those. tap_xy(_X, _Y) -> erlang:nif_error(not_loaded). type_text(_Text) -> erlang:nif_error(not_loaded). delete_backward() -> erlang:nif_error(not_loaded). @@ -329,6 +347,13 @@ swipe_xy(_X1, _Y1, _X2, _Y2) -> erlang:nif_error(not_loaded). %% scroll_info(Id) -> #{offset, content, viewport, max_offset, kind} | {error, Reason} %% scroll_to(Id, X, Y) -> ok | {error, Reason} screenshot(_Format, _Quality, _Scale) -> erlang:nif_error(not_loaded). +%% sample_region(X, Y, W, H) -> {ok, PixelW, PixelH, RGBA} | {error, Reason} +%% X/Y/W/H are window points (the element_frames/0 space); RGBA is +%% PixelW*PixelH*4 bytes of 8-bit RGBA at the native screen scale. +%% Reason :: empty_region | offscreen | no_window | capture_failed. +%% Pixel sampling is the only reliable way to verify a drawn colour on iOS 26 — +%% see Mob.Test.sample_color/2. iOS only; the stub raises on Android. +sample_region(_X, _Y, _W, _H) -> erlang:nif_error(not_loaded). scroll_info(_Id) -> erlang:nif_error(not_loaded). scroll_to(_Id, _X, _Y) -> erlang:nif_error(not_loaded). %% element_frames() -> JSON binary {"id":[x,y,w,h],...} of on-screen frames for diff --git a/test/mob/release_screenshot_test.exs b/test/mob/release_screenshot_test.exs index a24f4f8..ebda222 100644 --- a/test/mob/release_screenshot_test.exs +++ b/test/mob/release_screenshot_test.exs @@ -51,6 +51,21 @@ defmodule Mob.ReleaseScreenshotTest do "screenshot must sit behind a MOB_ENABLE_SCREENSHOT opt-in guard; got: #{guards["screenshot"]}" end + test "sample_region stays debug-only — it must not ride the screenshot opt-in" do + guards = registration_guards() + assert guards["sample_region"], "sample_region NIF not found in the registration table" + + # sample_region returns raw pixels of an arbitrary rect. Shipped in a release + # build it would let a caller reconstruct the screen region by region, which + # is exactly the capability MOB_ENABLE_SCREENSHOT exists to make a conscious + # opt-in. It is a dev-time colour-verification tool: keep it out of release. + refute guards["sample_region"] =~ "MOB_ENABLE_SCREENSHOT", + "sample_region must not be release-opt-in; guard was: #{guards["sample_region"]}" + + assert guards["sample_region"] =~ "!MOB_RELEASE", + "sample_region must stay behind `#if !MOB_RELEASE`; guard was: #{guards["sample_region"]}" + end + test "private synthetic-input NIFs stay strictly debug-only, never release-opt-in" do guards = registration_guards() diff --git a/test/mob/test_test.exs b/test/mob/test_test.exs index 7d8c720..e81cd19 100644 --- a/test/mob/test_test.exs +++ b/test/mob/test_test.exs @@ -10,41 +10,59 @@ defmodule Mob.TestTest do defp sample_tree do %{ type: :root, + class: nil, label: nil, value: nil, frame: {0.0, 0.0, 393.0, 852.0}, + bg_color: nil, + text_color: nil, children: [ %{ type: :window, + class: "UIWindow", label: nil, value: nil, frame: {0.0, 0.0, 393.0, 852.0}, + bg_color: 0xFF000000, + text_color: nil, children: [ %{ type: :scroll, + class: "SwiftUI.ScrollViewHost", label: nil, value: nil, frame: {0.0, 62.0, 393.0, 756.0}, + bg_color: nil, + text_color: nil, children: [ %{ type: :button, + class: "SwiftUI.CGDrawingView", label: "Roll Dice", value: nil, frame: {24.0, 416.0, 327.0, 53.5}, + bg_color: 0xFF2196F3, + text_color: 0xFFFFFFFF, children: [] }, %{ type: :text, + class: "SwiftUI.CGDrawingView", label: "Hello", value: nil, frame: {24.0, 480.0, 100.0, 24.0}, + bg_color: nil, + text_color: 0xDE000000, children: [] }, %{ type: :button, + class: "SwiftUI.CGDrawingView", label: "Roll again", value: nil, frame: {24.0, 520.0, 327.0, 53.5}, + bg_color: 0xFF2196F3, + text_color: 0xFFFFFFFF, children: [] } ] @@ -83,15 +101,31 @@ defmodule Mob.TestTest do test "leaves with no children still emit one entry" do tree = %{ type: :button, + class: "UIButton", label: "Solo", value: nil, frame: {0.0, 0.0, 1.0, 1.0}, + bg_color: nil, + text_color: nil, children: [] } assert [{[], node}] = M.flatten_tree(tree) assert node.label == "Solo" end + + test "keeps painted colours on every flattened entry" do + colours = + sample_tree() + |> M.flatten_tree() + |> Enum.map(fn {_p, n} -> {n.bg_color, n.text_color} end) + + # A styling regression that dropped every Box background would turn each + # of these into {nil, nil} — that is the whole point of surfacing them. + assert {0xFF2196F3, 0xFFFFFFFF} in colours + assert {nil, 0xDE000000} in colours + assert Enum.count(colours, fn {bg, _} -> bg == 0xFF2196F3 end) == 2 + end end describe "find_view (search semantics)" do @@ -115,15 +149,21 @@ defmodule Mob.TestTest do test "matches against value as well as label" do tree = %{ type: :root, + class: nil, label: nil, value: nil, frame: {0.0, 0.0, 1.0, 1.0}, + bg_color: nil, + text_color: nil, children: [ %{ type: :text_field, + class: "UITextField", label: "Name", value: "Roll-something", frame: {0.0, 0.0, 1.0, 1.0}, + bg_color: nil, + text_color: nil, children: [] } ] @@ -141,21 +181,95 @@ defmodule Mob.TestTest do end end - describe "tree shape normalization (Android JSON path)" do - # Mob.Test.view_tree/1 normalizes JSON-decoded trees (string keys, list frame) - # into the iOS map shape (atom keys, tuple frame). normalize_tree is private - # but exercised via the public API by sending a JSON binary through view_tree - # would require RPC — so we test the contract by mirroring the JSON shape and - # asserting the documented surface. + describe "normalize_view_tree/1 (Android JSON path)" do + defp android_json do + """ + {"type":"root","class":null,"label":null,"value":null,"frame":[0,0,393,852], + "bg_color":null,"text_color":null, + "children":[{"type":"button","class":"android.widget.Button", + "label":"Save","value":null, + "frame":[24,720,327,48], + "bg_color":4280391411,"text_color":4294967295, + "children":[]}]} + """ + end + + test "converts string keys, string type and list frame to the iOS shape" do + root = android_json() |> :json.decode() |> M.normalize_view_tree() + + assert root.type == :root + assert root.frame == {0.0, 0.0, 393.0, 852.0} + + [button] = root.children + assert button.type == :button + assert button.label == "Save" + assert button.class == "android.widget.Button" + assert button.frame == {24.0, 720.0, 327.0, 48.0} + end + + test "carries colours through as 0xAARRGGBB integers, matching iOS" do + [button] = + android_json() |> :json.decode() |> M.normalize_view_tree() |> Map.fetch!(:children) + + # JSON has no hex literal, so the bridge sends the same integer decimal. + assert button.bg_color == 0xFF2196F3 + assert button.text_color == 0xFFFFFFFF + end + + test "JSON null becomes nil, not :json.decode's :null atom" do + root = android_json() |> :json.decode() |> M.normalize_view_tree() + + # A node that paints nothing still carries the keys, so consumers can read + # node.bg_color on any node and compare against nil. + assert Map.has_key?(root, :bg_color) + assert root.bg_color == nil + assert root.text_color == nil + assert root.label == nil + assert root.value == nil + assert root.class == nil + end + + test "passes non-node terms (e.g. an error tuple) through untouched" do + assert M.normalize_view_tree({:error, :not_loaded}) == {:error, :not_loaded} + end + end + + describe "color_census/1" do + test "tallies painted colours by channel, ignoring nils" do + assert %{background: bg, text: text} = M.color_census(sample_tree()) - test "documented output frame shape is a 4-tuple of floats" do - {x, y, w, h} = sample_tree().frame - for v <- [x, y, w, h], do: assert(is_float(v)) + assert bg == %{0xFF000000 => 1, 0xFF2196F3 => 2} + assert text == %{0xFFFFFFFF => 2, 0xDE000000 => 1} end - test "documented output uses atom :type and :children keys" do - assert sample_tree().type == :root - assert length(sample_tree().children) > 0 + test "a tree whose backgrounds were all discarded reports none" do + stripped = strip_backgrounds(sample_tree()) + + assert %{background: bg, text: text} = M.color_census(stripped) + + # This is the regression shape the API exists to make visible: a theme + # that drops every Box background collapses :background to empty while + # text colours are untouched. + assert bg == %{} + assert map_size(text) == 2 + end + + test "two themes that differ produce different background key sets" do + primary = M.color_census(sample_tree()).background + raised = M.color_census(recolor(sample_tree(), 0xFF1E1E1E)).background + + refute primary == raised + assert Map.keys(raised) == [0xFF1E1E1E] + end + + defp strip_backgrounds(node), do: recolor(node, nil) + + defp recolor(%{children: children} = node, color) do + %{ + node + | bg_color: if(node.bg_color, do: color), + children: Enum.map(children, &recolor(&1, color)) + } end end @@ -257,4 +371,127 @@ defmodule Mob.TestTest do assert M.tour_offsets(info, []) == [{0.0, 0.0}] end end + + # ── Colour sampling (pure reduction over a raw RGBA buffer) ─────────────────── + + describe "reduce_rgba/3" do + # sample_region/4 hands back the buffer in R,G,B,A order; colours come out + # 0xAARRGGBB. `px` builds buffer bytes from a 0xAARRGGBB literal so the tests + # read in the repo's colour form. + defp px(argb) do + <> = <> + <> + end + + defp fill(argb, count), do: argb |> px() |> :binary.copy(count) + + test "channel order: R,G,B,A bytes in, 0xAARRGGBB out" do + # Distinct values per channel, so a swapped byte order can't pass. + assert {:ok, %{average: 0x04010203, dominant: 0x04010203}} = + M.reduce_rgba(<<1, 2, 3, 4>>, 1, 1) + end + + test "a single pixel reports itself with full confidence" do + assert {:ok, sample} = M.reduce_rgba(px(0xFF2196F3), 1, 1) + + assert sample == %{ + average: 0xFF2196F3, + dominant: 0xFF2196F3, + dominant_share: 1.0, + distinct: 1, + pixels: 1 + } + end + + test "a flat fill: average and dominant agree, share is 1.0" do + assert {:ok, sample} = M.reduce_rgba(fill(0xFF2196F3, 8), 4, 2) + + assert sample.average == 0xFF2196F3 + assert sample.dominant == 0xFF2196F3 + assert sample.dominant_share == 1.0 + assert sample.distinct == 1 + assert sample.pixels == 8 + end + + test "text over a fill: dominant keeps the background, average is dragged off it" do + # 8 px of :primary blue + 2 px of white "text". + buffer = fill(0xFF2196F3, 8) <> fill(0xFFFFFFFF, 2) + + assert {:ok, sample} = M.reduce_rgba(buffer, 5, 2) + + assert sample.dominant == 0xFF2196F3 + assert sample.dominant_share == 0.8 + assert sample.distinct == 2 + # (8*33 + 2*255)/10 = 77.4, (8*150 + 2*255)/10 = 171.0, (8*243 + 2*255)/10 = 245.4 + assert sample.average == 0xFF4DABF5 + refute sample.average == sample.dominant + end + + test "average is a per-channel mean rounded to nearest, alpha included" do + # Same RGB at alpha 0 and 255: alpha averages to round(127.5) = 128 and the + # colour channels are untouched — no un-premultiplying, no alpha weighting. + buffer = px(0x00808080) <> px(0xFF808080) + + assert {:ok, sample} = M.reduce_rgba(buffer, 2, 1) + assert sample.average == 0x80808080 + end + + test "a tie for dominant breaks toward the higher colour value (deterministic)" do + buffer = px(0x00808080) <> px(0xFF808080) + + assert {:ok, sample} = M.reduce_rgba(buffer, 2, 1) + assert sample.dominant == 0xFF808080 + assert sample.dominant_share == 0.5 + assert sample.distinct == 2 + end + + test "a gradient reports a low dominant_share — don't trust :dominant there" do + buffer = + Enum.map_join(0..9, fn i -> px(0xFF000000 + i) end) + + assert {:ok, sample} = M.reduce_rgba(buffer, 10, 1) + + assert sample.distinct == 10 + assert sample.dominant_share == 0.1 + end + + test "zero-size regions are refused, not reported as black" do + assert M.reduce_rgba(<<>>, 0, 0) == {:error, :empty_region} + assert M.reduce_rgba(<<>>, 10, 0) == {:error, :empty_region} + assert M.reduce_rgba(px(0xFF2196F3), -1, 1) == {:error, :empty_region} + end + + test "a buffer that doesn't match the reported dimensions yields no colour" do + # The whole point of this branch: a truncated or mis-sized buffer must not + # produce a plausible-looking colour from partial data. + assert M.reduce_rgba(fill(0xFF2196F3, 7), 4, 2) == {:error, :size_mismatch} + assert M.reduce_rgba(fill(0xFF2196F3, 9), 4, 2) == {:error, :size_mismatch} + assert M.reduce_rgba(<<1, 2, 3>>, 1, 1) == {:error, :size_mismatch} + end + + test "the glass-theme regression is visible in the reduction" do + # The real bug: the iOS renderer discarded every Box background, so a Box + # with background: :primary rendered identically to one with + # :surface_raised. Sampling the two regions is what makes that detectable. + primary = fill(0xFF2196F3, 16) + surface_raised = fill(0xFF1E1E1E, 16) + + {:ok, a} = M.reduce_rgba(primary, 4, 4) + {:ok, b} = M.reduce_rgba(surface_raised, 4, 4) + refute a.dominant == b.dominant + + # Under the bug both Boxes paint the theme's base surface: identical samples. + {:ok, bug_a} = M.reduce_rgba(surface_raised, 4, 4) + assert bug_a.dominant == b.dominant + end + end + + describe "sample_color/2" do + test "an unreachable node surfaces the dist failure instead of a colour" do + assert {:error, {:badrpc, _}} = M.sample_color(:"nonexistent_mob_node@127.0.0.1", "card") + + assert {:error, {:badrpc, _}} = + M.sample_color(:"nonexistent_mob_node@127.0.0.1", {0.0, 0.0, 10.0, 10.0}) + end + end end