Skip to content

Commit 5f39559

Browse files
committed
devbox ios ui drives the app on the device through WebDriverAgent, the way a test or an agent checks a change on a real iPhone: install <WebDriverAgentRunner-Runner.zip> signs and installs the runner of Appium like an app, start launches it through the developer disk image (pymobiledevice3 dvt xcuitest) and forwards its ports to 127.0.0.1 (8100, MJPEG 9100), then session, labels (rects in points), find <class>, tap <text>, tapxy, swipe, type, alert, activate, shot, record <seconds> <dir> (about 30 images a second, for a layout that moves), net <condition>|off|list (Network Link Conditioner through DVT), prefs <bundle id> (UserDefaults as JSON, Data holding JSON decoded), crashes and stop. devbox help ios-ui gives the steps and what was found on a real device: the accessibility press of pymobiledevice3 does nothing on an app signed by xtool, a sheet's field is found by its class, iOS keeps an app's preferences in memory so they are read only, a link opened in another app needs activate. Checked on an iPhone under iOS 27.2 with a Compose-free SwiftUI + Kotlin Multiplatform app: start, session, labels, tap, find, type, shot, record (27 frames in 1 s), net, prefs, crashes, alert; devbox help --check passes; shellcheck clean
1 parent fc082c1 commit 5f39559

6 files changed

Lines changed: 498 additions & 5 deletions

File tree

‎docs/commands.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ devbox commands # bare list, one name per line, for completions
3636
| `changelog` | `dev-box-changelog` | what changed in the box, newest first; the 3 latest, `-n N`, `--all`, `--upcoming` for what the next image brings |
3737
| `tour` | `dev-box-tour` | a guided tour of the box, two minutes, commands run under your eyes |
3838
| `desktop` | `dev-box-desktop` | the graphical session of `devbox dev-env desktop`, reached with gliff, started at connection and stopped by hand: in the menu and the list once `devbox dev-env desktop` is installed; a menu without an argument, `status`, `start`, `stop`, and its graphical apps (VS Code, Chromium): `apps`, `install`, `remove`, `open` |
39-
| `ios` | `dev-box-ios` | the iOS SDK of `devbox dev-env ios` in one go: a `.xip` of Xcode picked with gum among the ones of the home, `install-toolchain.sh` run on it in the right form, the SDK checked, the `.xip` deleted if you say so, then the Apple sign-in (run again, it only signs in); `sdk <xip>` without the list, `status`; `login` signs in to Apple alone, on a terminal (the options of `xtool auth login` pass through), `logout` signs out (`--reset-2fa` too); `devices` lists the devices seen over USB, `usbmuxd` started first (answer Trust on the device the first time); `pair` pairs the device on USB for Wi-Fi, once (iOS 27); `kmp [dir]` links the iOS framework of a Kotlin Multiplatform project on Linux, which Gradle skips, through `kmp/build-framework.sh` of omarchy-apple-dev (`--target ios_simulator_arm64`, `--module`, `--out`, `--dynamic`, `--xcframework`), also as root from the host for a project mounted with rootless Docker; `xcode [dir] [run|project|build|install|launch|logs]` builds the app of an Xcode project on Linux, then signs it for the device plugged in, or over Wi-Fi with `--wifi <address>` once paired (`IOS_WIFI`, `devbox help ios-wifi`), installs and launches it (`--project` for an `.xcodeproj` or an XcodeGen `project.yml`, `--target`, `--configuration`, `--package name=path[@ref]`, `--kmp :module` for a Kotlin Multiplatform app, `--signed-id`, `--signed-suffix`, `--out`, `--min-ios`, `--no-darwin-manifests`), also as root from the host. Needs the omarchy-apple-dev release the image pins: after an image update, `devbox dev-env ios` first (`status` shows it). In the menu and the list once `devbox dev-env ios` is installed |
39+
| `ios` | `dev-box-ios` | the iOS SDK of `devbox dev-env ios` in one go: a `.xip` of Xcode picked with gum among the ones of the home, `install-toolchain.sh` run on it in the right form, the SDK checked, the `.xip` deleted if you say so, then the Apple sign-in (run again, it only signs in); `sdk <xip>` without the list, `status`; `login` signs in to Apple alone, on a terminal (the options of `xtool auth login` pass through), `logout` signs out (`--reset-2fa` too); `devices` lists the devices seen over USB, `usbmuxd` started first (answer Trust on the device the first time); `pair` pairs the device on USB for Wi-Fi, once (iOS 27); `kmp [dir]` links the iOS framework of a Kotlin Multiplatform project on Linux, which Gradle skips, through `kmp/build-framework.sh` of omarchy-apple-dev (`--target ios_simulator_arm64`, `--module`, `--out`, `--dynamic`, `--xcframework`), also as root from the host for a project mounted with rootless Docker; `xcode [dir] [run|project|build|install|launch|logs]` builds the app of an Xcode project on Linux, then signs it for the device plugged in, or over Wi-Fi with `--wifi <address>` once paired (`IOS_WIFI`, `devbox help ios-wifi`), installs and launches it (`--project` for an `.xcodeproj` or an XcodeGen `project.yml`, `--target`, `--configuration`, `--package name=path[@ref]`, `--kmp :module` for a Kotlin Multiplatform app, `--signed-id`, `--signed-suffix`, `--out`, `--min-ios`, `--no-darwin-manifests`), also as root from the host; `ui <command>` drives the app on the device plugged in through WebDriverAgent, the way a test or an agent checks a change: `install <WebDriverAgentRunner-Runner.zip>` signs and installs the runner, `start` launches it and forwards its ports (8100, video 9100), `session <bundle id>`, `labels`, `find <class>`, `tap <text>`, `tapxy`, `swipe`, `type`, `alert`, `activate`, `shot`, `record <seconds> <dir>` (about 30 images a second), `net <condition>|off|list`, `prefs <bundle id>`, `crashes`, `stop` (`devbox help ios-ui`). Needs the omarchy-apple-dev release the image pins: after an image update, `devbox dev-env ios` first (`status` shows it). In the menu and the list once `devbox dev-env ios` is installed |
4040
| `help` | `dev-box-help` | step by step help for what a command cannot do alone: without a topic, a list to pick from; `--list`, `--check` |
4141
| `migrate` | `dev-box-migrate` | run the migrations shipped by the image, once each |
4242
| `mise-install` | `dev-box-mise-install` | write a mise-backed wrapper into `~/.local/bin` |
@@ -142,6 +142,7 @@ devbox help --check # hold every topic to the layout
142142
| `ios` | what is left to you after `devbox dev-env ios`: Xcode and the SDK, the Apple sign-in, the device, a first app, debugging and TestFlight, a newer Swift |
143143
| `ios-usb` | an iPhone or iPad plugged into your machine while the box runs on another |
144144
| `ios-wifi` | an iPhone or iPad under iOS 27 over Wi-Fi, without the cable: `devbox ios pair`, then `devbox ios xcode --wifi` |
145+
| `ios-ui` | drive and check an iOS app on the device from the box: WebDriverAgent installed once, then `devbox ios ui start`, `session`, `labels`, `tap`, `shot`, `record`, `net`, `stop`, and what was found on a real device |
145146
| `gpu` | giving `/dev/dri` to the box on the host, then the drivers and the test |
146147
| `desktop` | a graphical session of the box shown by gliff: the GPU, `devbox dev-env desktop`, connecting, the keys, stopping it |
147148
| `podman` | turning docker and podman on: the override block, `PODMAN_ENABLE`, the restart |

‎rootfs/usr/local/bin/dev-box-ios‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# devbox:name=ios
33
# devbox:summary=The iOS SDK from an Xcode .xip picked in a list, then the Apple sign-in, in one go
44
# devbox:summary.fr=Le SDK iOS depuis un .xip de Xcode choisi dans une liste, puis la connexion Apple, d'une traite
5-
# devbox:args=[sdk [xip]|status|login|logout|devices|pair|kmp [dir]|xcode [dir] [step]]
5+
# devbox:args=[sdk [xip]|status|login|logout|devices|pair|kmp [dir]|xcode [dir] [step]|ui <command>]
66
# devbox:requires=ios
77
#
88
# Step 1 of `devbox help ios`, chained: the .xip picked among the ones found
@@ -29,6 +29,10 @@
2929
# dev-box-ios xcode [dir] the app of an Xcode project, built on Linux,
3030
# installed and launched on the device plugged
3131
# in or over Wi-Fi with --wifi (xcode --help)
32+
# dev-box-ios ui <command> the app on the device driven through
33+
# WebDriverAgent: install, start, session, labels,
34+
# tap, type, shot, record, net, prefs, crashes,
35+
# stop (devbox help ios-ui)
3236
#
3337
# The SDK registered and its cache there: nothing to build. The cache there
3438
# and the SDK gone (a new Swift): registered again from the cache, no .xip.
@@ -54,7 +58,7 @@ WIFI_PAIRING="$HOME/.local/share/pymobiledevice3"
5458
# Where a .xip is looked for: the home and the projects, a few levels down.
5559
SEARCH_DEPTH=3
5660

57-
usage() { sed -n '8,38p' "$0" | sed 's/^# \?//'; }
61+
usage() { sed -n '8,42p' "$0" | sed 's/^# \?//'; }
5862
log() { printf '\033[1m[dev-box-ios]\033[0m %s\n' "$*"; }
5963
err() { echo "dev-box-ios: $*" >&2; }
6064

@@ -432,6 +436,16 @@ xcode)
432436
;;
433437
esac
434438
;;
439+
ui)
440+
shift
441+
# shellcheck source=/usr/share/devbox/lib/ios-common.sh
442+
. /usr/share/devbox/lib/ios-common.sh
443+
# shellcheck source=/usr/share/devbox/lib/ios-ui.sh
444+
. /usr/share/devbox/lib/ios-ui.sh
445+
ios_root_env
446+
need_ios
447+
ios_ui "$@"
448+
;;
435449
-h | --help) usage ;;
436450
*)
437451
err "unknown argument '$1'"
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
Drive and check an iOS app on the device, from the box
2+
3+
The way a test or an agent checks a change on a real iPhone: taps,
4+
texts, screenshots and video through WebDriverAgent (the XCUITest runner
5+
of Appium), network conditions, crash reports, the app's preferences.
6+
Over USB, after devbox help ios (SDK, sign-in, device trusted) and once
7+
the app is installed (devbox ios xcode ... install or run). Everything
8+
runs in the box; nothing to tap by hand once the runner is installed.
9+
10+
1. On your machine: the runner of WebDriverAgent
11+
Download WebDriverAgentRunner-Runner.zip (the runner for a real
12+
device, not a -Sim- one) from the latest release of
13+
https://github.com/appium/WebDriverAgent/releases
14+
and put it in the home or a project of the box.
15+
16+
2. In the box: install it, once a week with a free Apple ID
17+
It is signed and installed like an app (XTL-<team>.com.facebook.
18+
WebDriverAgentRunner.xctrunner), and takes one of the 3 apps a free
19+
Apple ID may have on a device; the signature lasts 7 days.
20+
devbox ios ui install ~/WebDriverAgentRunner-Runner.zip
21+
On the device, once: Settings, Privacy & Security, Developer Mode on;
22+
Settings, Developer, Enable UI Automation on; trust the developer in
23+
Settings, General, VPN & Device Management if asked.
24+
25+
3. In the box: start it, open the app
26+
The runner is launched through the developer disk image (mounted when
27+
needed), and its ports reach 127.0.0.1 of the box: 8100 (commands)
28+
and 9100 (video). Keep the device unlocked.
29+
devbox ios ui start
30+
devbox ios ui session <bundle id> launches the app, keeps a session
31+
32+
4. In the box: drive it
33+
devbox ios ui labels what is on screen, rects in points
34+
devbox ios ui tap "Valider" the element whose label has it
35+
devbox ios ui tapxy 195 785 a point (points, not pixels)
36+
devbox ios ui swipe 350 450 40 450 from one point to the other
37+
devbox ios ui find TextField fields labels leaves out
38+
devbox ios ui type "boulevard" into the focused field
39+
devbox ios ui alert accept a system alert (camera, location)
40+
devbox ios ui activate <bundle id> the app back after a link opened
41+
devbox ios ui shot /tmp/a.png screenshot
42+
devbox ios ui record 3 /tmp/rec ~30 images a second, for a jump
43+
devbox ios ui net SlowNetwork100PctLoss offline (net list, net off)
44+
devbox ios ui prefs <bundle id> the app's preferences, as JSON
45+
devbox ios ui crashes <executable> its crash reports
46+
A check: open a screen, shot or labels, compare. For a layout that
47+
moves, record while tapping into the screen, then follow the y of an
48+
element frame by frame.
49+
50+
5. In the box: stop it
51+
WebDriverAgent also listens on the local network of the device: stop
52+
it once the checks are done (the network condition goes with it).
53+
devbox ios ui stop
54+
55+
What was found on a real device
56+
The accessibility service of pymobiledevice3 lists the elements but
57+
its press does nothing on an app signed by xtool: WebDriverAgent is
58+
the way to tap. A sheet (a bottom sheet with a field) may be missing
59+
from labels: find TextField gives its rect, tapxy on it, then type.
60+
iOS keeps an app's preferences in memory (cfprefsd): a plist pushed
61+
in their place is overwritten by the app, so prefs is read only. An
62+
article or a link opened in another app takes the foreground: activate
63+
brings the app back. A screenshot every few hundred milliseconds misses
64+
an animation of a fraction of a second: record does not.
65+
66+
Sources
67+
https://github.com/appium/WebDriverAgent
68+
https://github.com/doronz88/pymobiledevice3
Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
"""devbox ios ui <command>: the app on the device driven through WebDriverAgent.
2+
3+
WebDriverAgent answers on http://127.0.0.1:8100 and streams the screen on
4+
:9100 once `devbox ios ui start` runs. Points, not pixels: a tap at 195 785
5+
is the middle of the bottom of a 390x844 screen (labels shows every rect).
6+
7+
session <bundle id> opens a session on the app, launched if needed;
8+
every command below uses it
9+
activate <bundle id> brings an installed app back to the foreground
10+
(a link opened in Safari or Chrome, for example)
11+
labels what is on screen: type, rect x,y wxh, label
12+
find <class> the elements of an XCUIElementType (TextField,
13+
Button...) with their rects, for what labels
14+
leaves out (a field inside a sheet)
15+
tap <text> [n] taps the n-th (0) element whose label or name
16+
contains <text>
17+
tapxy <x> <y> taps a point
18+
swipe <x1> <y1> <x2> <y2> drags from one point to the other
19+
type <text> types into the focused field (tap it first)
20+
alert [accept|dismiss] the text of the system alert shown, or answers it
21+
shot <file.png> screenshot, full resolution
22+
record <seconds> <dir> the screen at about 30 images a second (MJPEG,
23+
half size), one JPEG a frame named
24+
<n>-<time>.jpg: for an animation or a jump
25+
that screenshots, a second apart, miss
26+
27+
The session is kept in $IOS_UI_DIR/session (devbox ios ui sets it).
28+
"""
29+
import base64
30+
import json
31+
import os
32+
import sys
33+
import time
34+
import urllib.error
35+
import urllib.request
36+
37+
PORT = int(os.environ.get("IOS_UI_PORT", "8100"))
38+
MJPEG_PORT = int(os.environ.get("IOS_UI_MJPEG_PORT", "9100"))
39+
BASE = f"http://127.0.0.1:{PORT}"
40+
STATE = os.environ.get("IOS_UI_DIR", os.path.expanduser("~/.cache/dev-box/ios-ui"))
41+
SESSION_FILE = os.path.join(STATE, "session")
42+
43+
44+
def call(method, path, body=None, timeout=60, fail=True):
45+
data = json.dumps(body).encode() if body is not None else None
46+
request = urllib.request.Request(BASE + path, data=data, method=method, headers={"Content-Type": "application/json"})
47+
try:
48+
with urllib.request.urlopen(request, timeout=timeout) as response:
49+
return json.loads(response.read() or b"{}")
50+
except urllib.error.HTTPError as error:
51+
reply = json.loads(error.read() or b"{}")
52+
if not fail:
53+
return {}
54+
sys.exit(f"WebDriverAgent: {reply.get('value', {}).get('message', reply)}")
55+
56+
57+
def session():
58+
try:
59+
return open(SESSION_FILE).read().strip()
60+
except OSError:
61+
sys.exit("no session: devbox ios ui session <bundle id> first")
62+
63+
64+
def in_session(method, path, body=None, timeout=60):
65+
return call(method, f"/session/{session()}{path}", body, timeout).get("value")
66+
67+
68+
def element_id(element):
69+
return element.get("ELEMENT") or element.get("element-6066-11e4-a52e-4f735466cecf")
70+
71+
72+
def walk(node, out):
73+
if node.get("label") or node.get("name"):
74+
out.append(node)
75+
for child in node.get("children") or []:
76+
walk(child, out)
77+
78+
79+
def short_type(kind):
80+
kind = kind or ""
81+
return kind[len("XCUIElementType"):] if kind.startswith("XCUIElementType") else kind
82+
83+
84+
def rect_text(r):
85+
return f'{r.get("x", 0):>5},{r.get("y", 0):>5} {r.get("width", 0):>4}x{r.get("height", 0):<4}'
86+
87+
88+
def cmd_session(bundle_id):
89+
reply = call("POST", "/session", {"capabilities": {"alwaysMatch": {"bundleId": bundle_id, "shouldWaitForQuiescence": False}}}, timeout=120)
90+
sid = reply.get("sessionId") or reply.get("value", {}).get("sessionId")
91+
if not sid:
92+
sys.exit(f"no session: {reply}")
93+
os.makedirs(STATE, exist_ok=True)
94+
open(SESSION_FILE, "w").write(sid)
95+
# The MJPEG stream of record: close to 30 images a second, half size
96+
in_session("POST", "/appium/settings", {"settings": {"mjpegServerFramerate": 30, "mjpegScalingFactor": 50, "mjpegServerScreenshotQuality": 40}})
97+
print("session", sid, "on", bundle_id)
98+
99+
100+
def cmd_activate(bundle_id):
101+
in_session("POST", "/wda/apps/activate", {"bundleId": bundle_id})
102+
print("foreground:", bundle_id)
103+
104+
105+
def cmd_labels():
106+
nodes = []
107+
walk(call("GET", "/source?format=json")["value"], nodes)
108+
for n in nodes:
109+
print(f'{short_type(n.get("type")):<14} {rect_text(n.get("rect") or {})} {(n.get("label") or n.get("name"))[:90]}')
110+
111+
112+
def cmd_find(kind):
113+
kind = kind if kind.startswith("XCUIElementType") else "XCUIElementType" + kind
114+
for element in in_session("POST", "/elements", {"using": "class name", "value": kind}):
115+
rect = in_session("GET", f"/element/{element_id(element)}/rect")
116+
print(short_type(kind), rect_text(rect))
117+
118+
119+
def cmd_tap(text, index="0"):
120+
quoted = text.replace("\\", "\\\\").replace("'", "\\'")
121+
found = in_session("POST", "/elements", {"using": "predicate string", "value": f"label CONTAINS '{quoted}' OR name CONTAINS '{quoted}'"})
122+
if len(found) <= int(index):
123+
sys.exit(f"not on screen: {text} (devbox ios ui labels)")
124+
in_session("POST", f"/element/{element_id(found[int(index)])}/click", {})
125+
print("tapped:", text)
126+
127+
128+
def cmd_tapxy(x, y):
129+
in_session("POST", "/wda/tap", {"x": float(x), "y": float(y)})
130+
print("tapped", x, y)
131+
132+
133+
def cmd_swipe(x1, y1, x2, y2):
134+
in_session("POST", "/wda/dragfromtoforduration", {"fromX": float(x1), "fromY": float(y1), "toX": float(x2), "toY": float(y2), "duration": 0.2})
135+
print("swiped")
136+
137+
138+
def cmd_type(text):
139+
in_session("POST", "/wda/keys", {"value": list(text)})
140+
print("typed")
141+
142+
143+
def cmd_alert(answer=None):
144+
if answer is None:
145+
reply = call("GET", f"/session/{session()}/alert/text", fail=False)
146+
print(reply.get("value") or "no alert")
147+
elif answer in ("accept", "dismiss"):
148+
in_session("POST", f"/alert/{answer}", {})
149+
print("alert:", answer)
150+
else:
151+
sys.exit("alert [accept|dismiss]")
152+
153+
154+
def cmd_shot(path):
155+
open(path, "wb").write(base64.b64decode(call("GET", "/screenshot")["value"]))
156+
print(path)
157+
158+
159+
def cmd_record(seconds, out):
160+
os.makedirs(out, exist_ok=True)
161+
stream = urllib.request.urlopen(f"http://127.0.0.1:{MJPEG_PORT}", timeout=10)
162+
buf, count, end = b"", 0, time.time() + float(seconds)
163+
while time.time() < end:
164+
buf += stream.read(4096)
165+
start = buf.find(b"\xff\xd8")
166+
stop = buf.find(b"\xff\xd9", start + 2) if start != -1 else -1
167+
if start != -1 and stop != -1:
168+
with open(os.path.join(out, f"{count:04d}-{time.time():.3f}.jpg"), "wb") as frame:
169+
frame.write(buf[start:stop + 2])
170+
count += 1
171+
buf = buf[stop + 2:]
172+
print(count, "frames in", out)
173+
174+
175+
COMMANDS = {
176+
"session": cmd_session, "activate": cmd_activate, "labels": cmd_labels, "find": cmd_find,
177+
"tap": cmd_tap, "tapxy": cmd_tapxy, "swipe": cmd_swipe, "type": cmd_type, "alert": cmd_alert,
178+
"shot": cmd_shot, "record": cmd_record,
179+
}
180+
181+
if __name__ == "__main__":
182+
if len(sys.argv) < 2 or sys.argv[1] not in COMMANDS:
183+
sys.exit(__doc__)
184+
try:
185+
COMMANDS[sys.argv[1]](*sys.argv[2:])
186+
except TypeError:
187+
sys.exit(__doc__)

0 commit comments

Comments
 (0)