Skip to content
This repository was archived by the owner on Nov 9, 2026. It is now read-only.

Live screen viewer (iphonectl viewer) and WebView debugging over CDP - #1

Merged
Derrors merged 7 commits into
Derrors:mainfrom
TheSalaty:feat/live-viewer-and-webview-debugging
Nov 7, 2026
Merged

Derrors merged 7 commits into
Derrors:mainfrom
TheSalaty:feat/live-viewer-and-webview-debugging

Conversation

@TheSalaty

@TheSalaty TheSalaty commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two additions that make the toolkit nicer for people watching or testing alongside the agent, plus a small fix that lets them coexist with the MCP server.

1. Live viewer: iphonectl viewer
A desktop window (Python + Tk + Pillow, runs on the host) that shows the phone screen live from WebDriverAgent's MJPEG server and forwards input:

Input On the phone
Click / hold Tap / long press
Drag, mouse wheel Swipe, scroll
Keyboard Types into the focused field
Esc, right click, ⌂ Home Home button
✕ Close app Terminates the foreground app

The window is freely resizable. Gets ~24 fps over USB at 50 % resolution (configurable: --fps, --scale, --quality). iOS ignores system gestures such as the swipe up from the bottom edge when they are injected (tested over both WDA and CoreDevice HID), so the window offers buttons for them instead.

2. WebView debugging over CDP
The supervisor now also runs pymobiledevice3 webinspector cdp on 127.0.0.1:9222. Safari tabs and inspectable WebViews (Capacitor, Cordova, any WKWebView with isInspectable) become available over the Chrome DevTools Protocol. For hybrid apps this is much faster than the element tree: evaluating JavaScript in the page took 3–20 ms here, vs ~1.5 s for /source. Documented: the Web Inspector setting on the phone, and the Capacitor setting needed when building without Xcode (Capacitor's debug detection relies on a CAPACITOR_DEBUG Info.plist key that only Xcode fills in).

3. Supervisor and doctor

  • Port forwards are tracked per UDID and port (ensure_forward), so WDA (8100) and MJPEG (9100) restart independently. Before, pkill -f 'usbmux forward' would have taken down every forward.
  • New .env options MJPEG_PORT (default 9100) and CDP_PORT (default 9222); 0 turns either off. Both bind to 127.0.0.1 only, like WDA.
  • doctor gets a section reporting the stream and the number of inspectable pages (warnings only, both are optional).

4. MCP: join the active WDA session
WebDriverAgent has a single session, and POST /session ends the current one. wda.session() now reuses the session from /status and only creates one if there is none, or once to replace a stale one. Without this, the viewer and the MCP server would keep ending each other's sessions.

Design notes

  • src/iphone_viewer/ is split into pure, tested modules (mjpeg stream parser, geometry for window-to-point mapping and gesture classification, wda client) and the Tk UI in app.py. It is excluded from the container's iphone-mcp package (include = ["iphone_mcp*"]). iphonectl viewer runs it with PYTHONPATH=src, so there is nothing to install besides Tk and Pillow.
  • Phone actions from the UI go through a single worker thread, so typed characters keep their order and the UI never blocks.
  • All new text (code, CLI and doctor messages, docs) is in English. I left README.zh-CN.md and the Chinese docs untouched, so a Chinese translation of the new README sections is still open if you want one.

Testing

  • ruff check src tests scripts, pytest (53 passed, 18 new), and shellcheck -x container/*.sh bin/iphonectl scripts/*.sh all pass.
  • On a real iPhone 15 (iOS 27.2) with a Manjaro host: ran the published image with this branch's container/ scripts and iphone_mcp mounted in.
    • All three services come up.
    • Killing the MJPEG forward restarts only that one.
    • doctor reports the new checks.
    • iphonectl viewer streams and controls the phone.
    • The MCP server reuses the viewer's session (the active session ID stays the same).
  • I did not build the full image locally; the CI image job covers that.

🤖 Generated with Claude Code

TheSalaty and others added 4 commits November 7, 2026 00:10
The supervisor now keeps two more services alive next to the WDA forward:
- 127.0.0.1:9100: WebDriverAgent's MJPEG screen stream
- 127.0.0.1:9222: pymobiledevice3's Chrome DevTools Protocol bridge for
  Safari tabs and inspectable WebViews

Forwards are tracked per UDID and port, so restarting one no longer kills
the others. Both ports bind to 127.0.0.1 only and can be turned off with
MJPEG_PORT=0 / CDP_PORT=0. doctor reports whether the stream is up and how
many pages are inspectable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WebDriverAgent runs a single session and opening a new one ends the current
one. Reuse the session reported by /status, so the MCP server and other
clients (such as iphonectl viewer) can work side by side. A stale session is
still replaced once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Tk window on the host shows the MJPEG stream and turns input into
touches: click/hold/drag/wheel for tap, long press, swipe and scroll,
keyboard for typing, plus Home and Close app buttons for the system
gestures iOS does not accept from outside. The window is freely
resizable. Stream parsing, window-to-point mapping, gesture
classification and session handling are unit tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
English README only. Includes the Web Inspector setting on the phone and the
Capacitor setting needed when building without Xcode (e.g. with xtool).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TheSalaty
TheSalaty force-pushed the feat/live-viewer-and-webview-debugging branch from 0298bf1 to 11ce38d Compare November 6, 2026 23:10
Derrors and others added 3 commits November 7, 2026 09:16
Sessions: once a cached session went stale, the retry always opened a new
one, which ended the other client's session, so after a WebDriverAgent
restart the MCP server and the viewer kept ending each other's sessions.
Both now join the active session on retry too, and open one only if there
is none or the active one is the one that just went stale.

Viewer:
- Talk to WDA and the stream directly: urllib sends even 127.0.0.1 through
  HTTP_PROXY on the host.
- Keep the action worker, frame reader and frame polling alive after an
  unexpected error. Before, input or the picture stopped without any notice.
- A long status message no longer widens the window, which moved the screen
  under the pointer and made a click land on the wrong spot.
- Explain a missing display instead of showing a traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vices

The CDP bridge gives any process on the host access to the phone's Safari
tabs, including logged-in sites, and has no authentication or origin check,
so it is now opt-in (CDP_PORT=0) and the README warns about it.

doctor checks that the MJPEG port answers with a multipart stream: the
container uses host networking and 9100 is a common port, so a plain HTTP
200 from another service used to count as a working live view.

README: how to run the viewer on a desktop when the phone is on a headless
host. Also fixes the width of a line in the architecture diagram.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README.zh-CN.md gets the features, the "实时画面" and "调试 WebView"
sections, the viewer command and the system gesture limitation from the
English README. The detailed docs now cover the MJPEG and CDP ports
(security), the shared WebDriverAgent session and per-port forwards
(architecture), common viewer and CDP problems (troubleshooting) and
src/iphone_viewer/ (development). .env.example comments are in Chinese
like the rest of the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Derrors
Derrors merged commit 51cf20d into Derrors:main Nov 7, 2026
4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants