Skip to content
This repository was archived by the owner on Nov 9, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ PGID=1000

# WDA 端口,只监听 127.0.0.1
WDA_PORT=8100
# 实时画面的 MJPEG 流端口(iphonectl viewer 用),只监听 127.0.0.1;0 = 关闭
MJPEG_PORT=9100
# WebView 调试(Chrome DevTools Protocol)端口,只监听 127.0.0.1。默认关闭:开启后本机任何进程都能读取、
# 操作手机上已登录的 Safari 标签页。设成端口号(一般是 9222)开启
CDP_PORT=0
# 接了多台手机时指定要控制的那台;留空用第一台
IPHONE_UDID=
# WDA 不可用、且从没连上过 WDA 时,HID 后备路线换算坐标用的屏幕尺寸(点);一般留空
Expand Down
50 changes: 50 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ Claude Code / any MCP client
- **Runs unattended**: recovers from replugging, phone reboots, tunnel drops and WebDriverAgent crashes; re-signs WebDriverAgent with a free Apple ID before the 7-day profile expires
- **Alerts**: Telegram or Bark notifications when re-signing fails, the phone goes offline, or a passcode is needed after a reboot
- **Optional auto-unlock**: the phone can lock normally; every passcode digit is verified, and a wrong passcode disables auto-unlock after one attempt
- **Live viewer**: `iphonectl viewer` opens a desktop window with the phone's screen (up to ~25 fps over USB) that you control with mouse and keyboard, while the agent keeps working
- **Optional WebView debugging**: Safari tabs and inspectable WebViews (Capacitor, Cordova, your own `WKWebView`) are exposed over the Chrome DevTools Protocol, so tools can read the DOM and console directly instead of going through screenshots
- **Diagnostics**: `iphonectl doctor` checks every layer from USB to WebDriverAgent and tells you how to fix each failure

## Quick Start
Expand Down Expand Up @@ -109,12 +111,59 @@ To build the image from source instead of pulling it, set `IMAGE=iphone-control-

Screenshot pixels are tap coordinates, so the agent can tap what it sees. Games, maps and video players draw their own UI and have no element tree — use screenshots and coordinates there.

## Live Viewer

```bash
bin/iphonectl viewer
```

A window on the host shows the phone's screen live, streamed from WebDriverAgent's MJPEG server, and passes your input on to the phone. It runs next to the MCP server and shares its WebDriverAgent session, so you can watch the agent work or take over.

| Input | On the phone |
| --- | --- |
| Click / hold | Tap / long press |
| Drag, mouse wheel | Swipe, scroll |
| Keyboard | Types into the focused field |
| Esc, right click, **⌂ Home** | Home button |
| **✕ Close app** | Ends the foreground app, like swiping it away in the app switcher |

iOS ignores system gestures that do not come from a finger, such as the swipe up from the bottom edge, so the window has buttons for them. Resize it freely. Options such as frame rate and resolution: `iphonectl viewer --help`.

It needs Python 3.11+ with Tk and Pillow on the host: `sudo apt install python3-tk python3-pil.imagetk` (Debian/Ubuntu) or `sudo pacman -S tk python-pillow` (Arch).

If the phone is plugged into a headless host, run the viewer on your desktop from a checkout of this repository:

```bash
ssh -N -L 8100:127.0.0.1:8100 -L 9100:127.0.0.1:9100 your-host &
PYTHONPATH=src python3 -m iphone_viewer
```

## Debugging WebViews

This is off by default. Set `CDP_PORT=9222` in `.env` and run `iphonectl up`, and the container serves the phone's Safari tabs and inspectable WebViews over the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) on `127.0.0.1:9222`: `curl 127.0.0.1:9222/json` lists them, and anything that speaks CDP can attach to read the DOM, run JavaScript, or follow console output and network requests. For hybrid apps that is faster and more precise than the element tree or screenshots.

> [!WARNING]
> While it is on, any process on the host can read and script the phone's Safari tabs, including sites you are logged in to. The bridge has no authentication and does not check where a request comes from, so turn it on only while you need it and only on a host you trust.

1. On the phone, turn on **Settings → Apps → Safari → Advanced → Web Inspector**.
2. Your app's `WKWebView` must be inspectable (`isInspectable = true`, iOS 16.4+). Safari is always inspectable.

> [!TIP]
> **Capacitor apps built without Xcode** (for example with [xtool](https://github.com/xtool-org/xtool)): Capacitor only makes the WebView inspectable in debug builds, which it detects through a `CAPACITOR_DEBUG` Info.plist key that only Xcode's build settings fill in. Set it explicitly in `capacitor.config.ts`, for development builds only:
>
> ```ts
> ios: { webContentsDebuggingEnabled: !isReleaseBuild },
> ```

`iphonectl doctor` reports how many pages are inspectable. Both the bridge and the live stream listen on `127.0.0.1` only and have no authentication, like WebDriverAgent itself; set `MJPEG_PORT=0` in `.env` to turn the stream off. To reach them from another machine, use SSH port forwarding rather than binding them to other interfaces.

## Everyday Commands

| Command | What it does |
| --- | --- |
| `iphonectl status` | Container status and recent supervisor log |
| `iphonectl doctor` | Layer-by-layer health check with a fix for each failure |
| `iphonectl viewer` | Live view of the screen, controlled with mouse and keyboard |
| `iphonectl logs [name] [-f]` | Logs: `supervisor`, `wda`, `tunnel`, `resign`, `notify`, … |
| `iphonectl resign [--force]` | Re-sign WebDriverAgent if it expires soon (or now) |
| `iphonectl login` | Sign in to the Apple ID again when the session expires |
Expand All @@ -140,6 +189,7 @@ The detailed docs are in Chinese for now.
- Free Apple ID profiles last 7 days. Re-signing is automatic, but an expired Apple ID session needs `iphonectl login`
- After the phone reboots, you must enter the passcode once on the phone to allow UI automation — iOS requires this
- Elements hidden behind a popup may still be listed, because visibility is judged by position
- System gestures (home indicator swipe, Control Center, Notification Center) cannot be injected from outside; use the Home button, closing apps by bundle ID, and URL schemes instead

> [!WARNING]
> This toolkit can fully control your phone. An agent may tap payment, delete or send buttons — keep tool-call confirmation on in your MCP client. Not affiliated with Apple; use it only with devices you own.
Expand Down
50 changes: 50 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ Claude Code / 任意 MCP 客户端
- **无人值守**:USB 重插、手机重启、隧道断开、WebDriverAgent 崩溃后都能自动恢复;免费 Apple ID 的描述文件 7 天到期前自动重签
- **通知**:重签失败、手机离线、重启后需要输入密码时,推送 Telegram 或 Bark
- **可选自动解锁**:手机可以正常锁屏;每一位密码都会核对,密码配置错误时输错一次就停用自动解锁
- **实时画面**:`iphonectl viewer` 在桌面上打开一个窗口,实时显示手机屏幕(USB 下最高约 25 fps),可以用鼠标和键盘操作,AI 照常工作
- **可选的 WebView 调试**:通过 Chrome DevTools Protocol 开放 Safari 标签页和可检查的 WebView(Capacitor、Cordova、你自己的 `WKWebView`),工具可以直接读 DOM 和控制台,不用靠截图
- **诊断**:`iphonectl doctor` 从 USB 到 WebDriverAgent 逐层检查,每项失败都给出修复办法

## 快速开始
Expand Down Expand Up @@ -109,12 +111,59 @@ bin/iphonectl mcp install # 注册到 Claude Code

截图的像素坐标就是点击坐标,AI 看到哪里就能点哪里。游戏、地图、视频播放器自己绘制界面,没有元素树,只能看截图按坐标操作。

## 实时画面

```bash
bin/iphonectl viewer
```

在宿主机上打开一个窗口,实时显示手机屏幕(来自 WebDriverAgent 的 MJPEG 流),并把你的操作转发给手机。它和 MCP 服务同时运行、共用同一个 WebDriverAgent 会话,你可以看着 AI 操作,也可以随时接手。

| 操作 | 手机上的效果 |
| --- | --- |
| 单击 / 按住 | 点击 / 长按 |
| 拖动、滚轮 | 滑动、滚动 |
| 键盘 | 在当前输入框里输入 |
| Esc、右键、**⌂ Home** | Home 键 |
| **✕ Close app** | 结束前台 App,相当于在多任务界面里把它划掉 |

iOS 会忽略不是手指发出的系统手势,比如从屏幕底部上滑,所以窗口提供了对应的按钮。窗口大小可以随意调整。帧率、分辨率等选项见 `iphonectl viewer --help`。

宿主机需要 Python 3.11+,以及 Tk 和 Pillow:`sudo apt install python3-tk python3-pil.imagetk`(Debian/Ubuntu)或 `sudo pacman -S tk python-pillow`(Arch)。

手机接在没有图形界面的主机(比如 NAS)上时,在桌面电脑上克隆本仓库后运行:

```bash
ssh -N -L 8100:127.0.0.1:8100 -L 9100:127.0.0.1:9100 你的主机 &
PYTHONPATH=src python3 -m iphone_viewer
```

## 调试 WebView

默认关闭。在 `.env` 里设置 `CDP_PORT=9222` 后运行 `iphonectl up`,容器就会在 `127.0.0.1:9222` 上通过 [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) 开放手机的 Safari 标签页和可检查的 WebView:`curl 127.0.0.1:9222/json` 列出它们,任何支持 CDP 的工具都可以连上去读 DOM、执行 JavaScript、查看控制台输出和网络请求。对混合开发的 App 来说,这比元素树和截图更快也更准。

> [!WARNING]
> 开启期间,宿主机上的任何进程都能读取和操作手机的 Safari 标签页,包括你已登录的网站。桥接没有认证,也不检查请求来源,请只在需要时开启,并且只在你信任的主机上开启。

1. 在手机上打开 **设置 → App → Safari 浏览器 → 高级 → 网页检查器**。
2. App 里的 `WKWebView` 必须可检查(`isInspectable = true`,iOS 16.4+)。Safari 始终可检查。

> [!TIP]
> **不用 Xcode 构建的 Capacitor App**(比如用 [xtool](https://github.com/xtool-org/xtool)):Capacitor 只在调试构建里让 WebView 可检查,而它是通过 Info.plist 里的 `CAPACITOR_DEBUG` 判断调试构建的,这个键只有 Xcode 的构建设置会填。请在 `capacitor.config.ts` 里显式设置,并且只用于开发构建:
>
> ```ts
> ios: { webContentsDebuggingEnabled: !isReleaseBuild },
> ```

`iphonectl doctor` 会报告有几个可检查的页面。CDP 桥接和实时画面都只监听 `127.0.0.1`,和 WebDriverAgent 一样没有认证;不需要实时画面时,在 `.env` 里设置 `MJPEG_PORT=0` 关闭。要从其他机器访问,请用 SSH 端口转发,不要把它们绑到其他网卡上。

## 常用命令

| 命令 | 作用 |
| --- | --- |
| `iphonectl status` | 容器状态和最近的守护日志 |
| `iphonectl doctor` | 逐层诊断,每项失败都给出修复办法 |
| `iphonectl viewer` | 实时查看屏幕,用鼠标和键盘操作 |
| `iphonectl logs [名字] [-f]` | 看日志:`supervisor`、`wda`、`tunnel`、`resign`、`notify` 等 |
| `iphonectl resign [--force]` | 快过期时重签 WebDriverAgent;`--force` 立即重签 |
| `iphonectl login` | Apple ID 登录过期时重新登录 |
Expand All @@ -138,6 +187,7 @@ bin/iphonectl mcp install # 注册到 Claude Code
- 免费 Apple ID 的描述文件 7 天过期。重签是自动的,但 Apple ID 登录过期后要运行 `iphonectl login`
- 手机重启后,要在手机上输一次锁屏密码才能启用 UI 自动化,这是 iOS 的要求
- 可见性按坐标判断,被弹窗挡住的元素也可能被列出来
- 系统手势(底部上滑、控制中心、通知中心)无法从外部注入;请改用 Home 键、按 Bundle ID 结束 App 和 URL Scheme

> [!WARNING]
> 这个工具能完全控制你的手机。AI 可能点到付款、删除、发送等按钮,请在 MCP 客户端里保持工具调用确认。本项目与 Apple 无关,请只用于你自己的设备。
Expand Down
12 changes: 11 additions & 1 deletion bin/iphonectl
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ cmd_shell() { need_running; inct bash; }
cmd_login() { need_running; inct plumesign account login "$@"; }
cmd_resign() { need_running; inc /opt/ictl/resign-wda.sh "$@"; }
cmd_notify_test() { need_running; inc /opt/ictl/notify.sh --test; }

# Live view window. Runs on the host (needs Python with Tk and Pillow) and drives the phone through WDA and the MJPEG port on 127.0.0.1
cmd_viewer() {
need_running
local mjpeg; mjpeg=$(env_get MJPEG_PORT); mjpeg=${mjpeg:-9100}
[ "$mjpeg" -gt 0 ] || die "The live view is turned off (MJPEG_PORT=0 in .env)"
PYTHONPATH="$ROOT/src${PYTHONPATH:+:$PYTHONPATH}" exec python3 -m iphone_viewer \
--wda "http://127.0.0.1:$PORT" --mjpeg "http://127.0.0.1:$mjpeg" "$@"
}
cmd_unlock_reset() { need_running; inc rm -f /data/logs/UNLOCK-DISABLED; say "已恢复自动解锁"; }

cmd_status() {
Expand Down Expand Up @@ -198,6 +207,7 @@ usage() {
passcode set|clear 设置/删除锁屏密码(自动解锁)
unlock-reset 自动解锁因输错被停用后,核对密码再恢复
mcp [install] 显示/执行注册到 Claude Code 的命令
viewer [options] Live view of the screen, controlled with mouse and keyboard (needs python3-tk and Pillow; --help)

容器
up | down | restart | pull | build | shell
Expand All @@ -210,7 +220,7 @@ EOF

cmd=${1:-help}; shift || true
case $cmd in
setup|up|down|restart|pull|build|status|doctor|logs|resign|login|shell|passcode|mcp)
setup|up|down|restart|pull|build|status|doctor|logs|resign|login|shell|passcode|mcp|viewer)
"cmd_$cmd" "$@" ;;
notify-test) cmd_notify_test ;;
unlock-reset) cmd_unlock_reset ;;
Expand Down
22 changes: 22 additions & 0 deletions container/doctor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ warn() { echo "⚠️ $1${2:+ $2}"; }
bad() { echo "❌ $1${2:+ $2}"; [ -n "${3:-}" ] && echo " → $3"; fails=$((fails + 1)); }
skip() { echo "⏭ $1 (跳过:$blocked)"; }
check() { [ -z "$blocked" ]; }
off() { echo "⏭ $1 (off: $2=0)"; }

echo "== 设备连接"
if pgrep -x usbmuxd >/dev/null; then ok "usbmuxd 在运行"; else
Expand Down Expand Up @@ -83,6 +84,27 @@ if check && [ -n "$bundle" ]; then
fi
else skip "WDA"; fi

echo "== Live view and WebView debugging"
if [ "$MJPEG_PORT" -eq 0 ]; then off "Live view" MJPEG_PORT
elif check && [ -n "$bundle" ]; then
# the MJPEG stream never ends, so curl always times out; the headers tell it apart from another service on the port
head=$(curl -s -m 2 -o /dev/null -w '%{http_code} %{content_type}' "http://127.0.0.1:$MJPEG_PORT")
case $head in
"200 multipart/"*) ok "Live view" "http://127.0.0.1:$MJPEG_PORT, open it on the host with iphonectl viewer" ;;
*) warn "Live view unavailable" "served by WDA, so only once WDA is ready; is another service on port $MJPEG_PORT? see data/logs/forward.log" ;;
esac
else skip "Live view"; fi

if [ "$CDP_PORT" -eq 0 ]; then off "WebView debugging" CDP_PORT
elif check; then
if pages=$(curl -sf -m 5 "http://127.0.0.1:$CDP_PORT/json"); then
n=$(printf '%s' "$pages" | grep -o '"webSocketDebuggerUrl"' | wc -l)
if [ "$n" -gt 0 ]; then ok "WebView debugging" "$n inspectable page(s), listed at http://127.0.0.1:$CDP_PORT/json"
else warn "WebView debugging: no inspectable pages" \
"turn on Settings → Apps → Safari → Advanced → Web Inspector; an app's WKWebView needs isInspectable"; fi
else warn "WebView debugging unavailable" "see data/logs/cdp.log"; fi
else skip "WebView debugging"; fi

echo "== 附加功能"
if [ -n "${TG_BOT_TOKEN:-}${BARK_KEY:-}" ]; then ok "通知已配置" "测试:iphonectl notify-test"; else
warn "没有配置通知" "重签失败、手机离线、需要输密码时不会提醒你"; fi
Expand Down
4 changes: 4 additions & 0 deletions container/lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ STATE_DIR=$DATA/state
SECRETS_DIR=$DATA/secrets
PLUME_DIR=$DATA/plume
WDA_PORT=${WDA_PORT:-8100}
# Live screen (WDA's MJPEG stream) and WebView debugging (Chrome DevTools Protocol) ports, 127.0.0.1 only; 0 = off.
# CDP stays off unless asked for: it gives access to the phone's logged-in Safari tabs
MJPEG_PORT=${MJPEG_PORT:-9100}
CDP_PORT=${CDP_PORT:-0}
WDA_UNSIGNED=/opt/ictl/wda/WDA-unsigned.ipa
PROFILE=$CERT_DIR/free-dev.mobileprovision
PUID=${PUID:-1000}
Expand Down
26 changes: 20 additions & 6 deletions container/supervisor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,23 @@ start_tunnel() {
log "隧道已(重新)启动"
}
# 不用 ios forward:它监听所有网卡,而 WDA 没有认证;pymobiledevice3 的转发默认只绑 127.0.0.1
start_forward() {
pkill -f 'usbmux forward'; sleep 1
pymobiledevice3 usbmux forward --udid "$UDID" "$WDA_PORT" "$WDA_PORT" >> "$LOG_DIR/forward.log" 2>&1 &
fwd_udid=$UDID
log "端口转发 127.0.0.1:$WDA_PORT 已启动"
# One forward process per port, identified by UDID and port: restarted when the phone changes, independent of each other.
ensure_forward() {
[ "$1" -gt 0 ] || return 0
running "usbmux forward --udid $UDID $1 $1\$" && return 0
pkill -f "usbmux forward --udid .* $1 $1\$"; sleep 1
pymobiledevice3 usbmux forward --udid "$UDID" "$1" "$1" >> "$LOG_DIR/forward.log" 2>&1 &
log "Port forward 127.0.0.1:$1 started"
}

# WebView debugging: serves the phone's Safari tabs and inspectable WebViews over the Chrome DevTools Protocol.
# Needs Settings → Apps → Safari → Advanced → Web Inspector on the phone, and isInspectable on an app's WKWebView.
ensure_cdp() {
[ "$CDP_PORT" -gt 0 ] || return 0
running "webinspector cdp --udid $UDID " && return 0
pkill -f 'webinspector cdp'; sleep 1
pymobiledevice3 webinspector cdp --udid "$UDID" --host 127.0.0.1 --port "$CDP_PORT" >> "$LOG_DIR/cdp.log" 2>&1 &
log "CDP bridge 127.0.0.1:$CDP_PORT started"
}
# 手机重启后开发者镜像会掉;已挂载时 auto-mount 约 1 秒直接跳过
start_wda() {
Expand Down Expand Up @@ -105,7 +117,9 @@ while :; do
fi
fi

if ! running 'usbmux forward' || [ "${fwd_udid:-}" != "$UDID" ]; then start_forward; fi
ensure_forward "$WDA_PORT"
ensure_forward "$MJPEG_PORT"
ensure_cdp

bundle=$(wda_bundle)
if [ -z "$bundle" ]; then
Expand Down
Loading
Loading