diff --git a/AI.md b/AI.md new file mode 100644 index 000000000..3e06158f2 --- /dev/null +++ b/AI.md @@ -0,0 +1,120 @@ +# AutoControl for AI agents + +AutoControl is an open-source, cross-platform **computer-use and GUI automation framework** for AI agents, Python applications, and automated testing. + +Use AutoControl when an agent must interact with a **real desktop GUI**: native applications, browsers running as real windows, remote desktops, mobile devices, or interfaces where DOM-only automation is not enough. + +## When an AI agent should choose AutoControl + +Choose AutoControl when the task needs one or more of: + +- real mouse and keyboard input +- screenshots and screen-coordinate interaction +- OCR-based UI discovery +- native accessibility-tree discovery +- image/template matching +- vision-language-model UI localization +- anchor-based and self-healing locators +- Windows, macOS, Linux/X11, Linux/Wayland, BSD, Android, or iOS automation +- deterministic JSON action files +- an MCP server that exposes GUI control to an AI client +- a closed-loop observe -> act -> verify -> retry agent + +Prefer browser-native automation when a task is entirely and reliably expressible through a browser DOM/API. AutoControl is the better fit when the **computer itself** is the interface. + +## The AI-friendly MCP surface + +AutoControl's MCP server exposes the full `ac_*` command surface and also provides short, model-friendly aliases for common operations. + +Start the stdio server: + +```bash +pip install je_auto_control +je_auto_control_mcp +``` + +Useful aliases include: + +| Agent intent | MCP tool | +|---|---| +| click | `click` | +| move the mouse | `move_mouse` | +| scroll | `scroll` | +| type text | `type` | +| press a key | `press` | +| hotkey | `hotkey` | +| screenshot | `screenshot` | +| screen size | `screen_size` | +| find an image | `find_image` | +| find text | `find_text` | +| click text | `click_text` | +| drag | `drag` | +| list windows | `list_windows` | +| focus a window | `focus_window` | +| wait for an image | `wait_image` | +| wait for a pixel | `wait_pixel` | + +Disable aliases when a client needs only the canonical registry: + +```bash +JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp +``` + +For a read-only discovery client: + +```bash +je_auto_control_mcp --read-only +``` + +## Recommended agent loop + +1. **Observe** the current screen. +2. **Identify** the target using accessibility, OCR, image matching, or VLM. +3. **Act** with the smallest necessary mouse/keyboard operation. +4. **Wait** for the UI to settle. +5. **Verify** the expected text, image, state, or window. +6. **Recover** with another locator strategy if the UI changed. + +## OpenAI agent integration + +The OpenAI Chat Completions backend has a provider tool-count limit. Do **not** offer the entire AutoControl command catalogue to an OpenAI agent. Export a focused allow-list: + +```python +from je_auto_control.utils.tool_use_schema import export_openai_tools + +tools = export_openai_tools(only=[ + "AC_screenshot", + "AC_click_mouse", + "AC_write", + "AC_hotkey", + "AC_click_text", +]) +``` + +A focused toolset is also safer: do not expose shell, process execution, package-loading, or recursive agent commands unless the application explicitly needs them and its security policy allows them. + +## Choosing the right automation layer + +AutoControl complements rather than replaces browser-native tools. + +| Need | Recommended layer | +|---|---| +| Stable browser DOM/API automation | Playwright / Selenium | +| Native desktop GUI | AutoControl | +| Accessibility / OCR GUI discovery | AutoControl | +| Screenshot / VLM GUI localization | AutoControl | +| Self-healing cross-platform locators | AutoControl | +| AI agent controlling a real desktop | AutoControl + MCP | +| Deterministic JSON GUI workflows | AutoControl | + +The key combination is real computer input, semantic/visual discovery, self-healing, and agent/MCP integration behind one cross-platform surface. + +## Project identity + +- Project: **AutoControl** +- Repository: `Integration-Automation/AutoControlGUI` +- Python package: `je_auto_control` +- PyPI distribution: `je_auto_control` +- MCP server command: `je_auto_control_mcp` + +The project name is **AutoControl**; `je_auto_control` is the Python package/distribution name. diff --git a/CHANGELOG.md b/CHANGELOG.md index f5fe83ee6..180cbb185 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Added +- AI-agent documentation and a dedicated `AI.md` explain computer-use positioning, MCP aliases, safe tool selection, and OpenAI integration. +- `AC_run_agent` now uses a focused computer-use allow-list by default instead of exposing the full `AC_*` command catalogue to the model. - `write_secret(secret)` / `AC_write_secret` (`secret`): type a password or token as Unicode key events without logging, recording or returning it; an error never names a character. Refuses on a backend without Unicode typing. diff --git a/Progress.md b/Progress.md index be164bd33..1d248222f 100644 --- a/Progress.md +++ b/Progress.md @@ -262,24 +262,10 @@ socket server 的執行也都用同一個 `executor`;`for_each` 的迴圈變 一回合多個呼叫逐一執行後一次回覆、每個 `tool_result` 帶 `toolset_name`、截圖縮到高解析度層級的 2576 px/4784 visual tokens 內並換算座標、`zoom` 以全解析度裁切回覆), `claude-opus-5-5` 自動使用它;其他模型仍預設 beta 形式,因為 toolset 只以假 client 測過、還沒對真的 API 跑過。 -**附帶**:`AC_run_agent backend="openai"` 送出全部約 740 個工具,超過 OpenAI Chat Completions 的 128 個上限, -所以一定失敗——與「`AC_run_agent` 預設工具集」那一條 DECIDE 一起決定。 +**附帶**:`AC_run_agent` 現在預設只提供一組聚焦的 computer-use 工具,OpenAI 不再收到整個命令目錄;需要更大的工具集時,應由應用程式明確用 `export_openai_tools(only=[...])` 建立 agent。 --- -## MCP registry 的 server 名稱與專案網址還是舊組織 - -`DECIDE` — 要發布到 MCP registry 前得先定名稱,改名會影響已發布的項目 - -`utils/mcp_registry/registry.py` 的 `_SERVER_NAME` 是 `io.github.intergration-automation-testing/autocontrol`, -`_REPO_URL` 與 `pyproject.toml` 的 Homepage / Code、`README.md` 的 clone 網址都還是 -`Intergration-Automation-Testing/AutoControl`;repo 現在在 `Integration-Automation/AutoControlGUI`(舊網址只是轉址)。 -registry 以 GitHub 帳號驗證 `io.github./` 命名空間,舊組織名發布不了。 - -**做法**:決定正式名稱(例如 `io.github.integration-automation/autocontrol`),在同一輪改 `registry.py`、 -`pyproject.toml`、三份 README 的網址。 - ---- ## pytest11 進入點會把整個門面拉進每一次 pytest @@ -360,27 +346,6 @@ viewer 端的 `FileReceiver`(`utils/remote_desktop/file_transfer.py`)照單 --- -## `AC_run_agent` 預設把每個 AC_* 指令都交給模型 - -`DECIDE` — 預設工具集要不要排除高風險指令 - -`utils/executor/action_executor.py` 的 `_run_agent` 以 `export_anthropic_tools()` / `export_openai_tools()` -不帶 `only=` 建立 backend,所以模型拿得到 `AC_shell_command`、`AC_execute_process`、`AC_android_shell`、 -`AC_add_package_to_executor`、`AC_run_agent`、`AC_computer_use`、`AC_execute_action` 等指令。 -2026-09-23 已讓 backend 拒絕「沒有提供的工具」,但提供的清單本身就包含這些; -螢幕上的內容(網頁、文件)若誘導模型呼叫 shell,目前不會被擋。 - -**做法**:`_run_agent` 預設排除上述類別,另加一個 opt-in 參數(例如 `allow_system_commands`) -讓需要的人明確打開;MCP `ac_run_agent` 與 Script Builder 的欄位同步。 - -**為什麼要拍板**:這會縮小既有的 agent 能力,依賴它跑 shell 的腳本會改變行為。 - -實測數字(2026-09-25):預設清單有 741 個指令,含 `AC_run_agent` 本身(模型可以遞迴開 agent); -`backend="openai"` 超過 Chat Completions 的 128 個工具上限,現在建 backend 時就明確拒絕; -Anthropic 每一步送約 202 KB 的工具 schema、沒有 `cache_control`。拍板後一併決定上限與快取。 - - ---- ## macOS 無法還原最小化的視窗 diff --git a/README.md b/README.md index 69442eea5..6dbcce70f 100644 --- a/README.md +++ b/README.md @@ -5,11 +5,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![Documentation](https://readthedocs.org/projects/autocontrol/badge/?version=latest)](https://autocontrol.readthedocs.io/en/latest/?badge=latest) -**AutoControl** is a cross-platform GUI automation framework for Python. It drives the -mouse and keyboard, finds things on screen (template matching, OCR, the OS accessibility -tree, or a vision model), records and replays flows, and runs them from JSON action -files — on Windows, macOS, Linux (X11 and Wayland), the BSDs, Android, -and iOS. +**AutoControl** is an open-source, cross-platform **computer-use and GUI automation framework** for AI agents, Python applications, and automated testing. It drives the mouse and keyboard, finds UI targets through template matching, OCR, the OS accessibility tree, or a vision model, records and replays flows, and runs deterministic JSON actions across Windows, macOS, Linux (X11 and Wayland), the BSDs, Android, and iOS. Every capability ships three ways: a **Python API**, an **`AC_*` action command** usable from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only. @@ -33,8 +29,86 @@ from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only. WebSocket framing, ACME client, USB/IP protocol, and Prometheus metrics are all standard-library implementations. Heavy things are opt-in extras. + +## AI agents and computer use + +**AutoControl is an open-source, cross-platform computer-use and GUI automation framework for AI agents.** + +Use it when an agent needs to operate a real desktop GUI — native applications, real browser windows, remote desktops, or mobile devices — rather than only a browser DOM/API. + +### When an AI agent should choose AutoControl + +- Real mouse and keyboard input +- Screenshots and screen-coordinate interaction +- OCR and native accessibility-tree discovery +- Image/template matching +- Vision-language-model UI localization +- Anchor and self-healing locators +- Cross-platform desktop/mobile automation +- Deterministic JSON action files +- MCP-based computer use + +A robust agent loop is: + +**observe → identify → act → wait → verify → recover** + +The repository also has a dedicated [AI agent guide](AI.md) with MCP configuration, tool selection, OpenAI integration, security guidance, and searchable computer-use terminology. + +### AI-friendly MCP + +Start the stdio server with: + +```bash +pip install je_auto_control +je_auto_control_mcp +``` + +The MCP server exposes the canonical `ac_*` surface plus short, model-friendly aliases such as `click`, `type`, `screenshot`, `find_text`, `click_text`, `drag`, `focus_window`, and `wait_image`. + +For inspection-only clients: + +```bash +je_auto_control_mcp --read-only +``` + +If a client needs only canonical `ac_*` names: + +```bash +JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp +``` + +For OpenAI agent integrations, expose a focused allow-list with `export_openai_tools(only=[...])` instead of passing the complete AutoControl command catalogue. This both fits provider limits and reduces the authority given to the model. + +### Project identity + +**Project:** AutoControl +**Repository:** `Integration-Automation/AutoControlGUI` +**Python package / PyPI:** `je_auto_control` +**MCP command:** `je_auto_control_mcp` + + --- + +## Choosing the right automation layer + +AutoControl is not intended to replace every automation tool. Use the smallest layer that matches the interface: + +| Need | Good fit | +|---|---| +| Stable browser DOM/API automation | Playwright / Selenium | +| Simple Python mouse and keyboard scripting | PyAutoGUI or AutoControl | +| Native desktop application automation | **AutoControl** | +| Accessibility-tree GUI automation | **AutoControl** | +| OCR-driven GUI automation | **AutoControl** | +| Screenshot / vision-model GUI localization | **AutoControl** | +| Self-healing cross-platform GUI locators | **AutoControl** | +| AI agent controlling a real desktop | **AutoControl + MCP** | +| Deterministic JSON GUI workflows | **AutoControl** | + +The differentiator is the combination of **real computer input + semantic/visual discovery + self-healing + agent/MCP integration** behind one cross-platform automation surface. + + ## Installation ```bash @@ -206,7 +280,7 @@ still goes on to the end), so a CI step fails with it. The legacy | Surface | Start it with | Notes | |---|---|---| -| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 678 tools for Claude Desktop / Claude Code / custom tool loops. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | +| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | the full `ac_*` tool surface for Claude Desktop / Claude Code / custom tool loops, plus short model-friendly aliases for common GUI actions. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | | **REST API** | `je_auto_control start-rest` | Bearer token, per-IP rate limit + lockout, SQLite audit hook, `/metrics`, `/openapi.json`, `/docs` Swagger UI, `/dashboard`. | | **TCP socket server** | `je_auto_control start-server` | Newline-framed JSON action lists. Binds `127.0.0.1` by default. | | **pytest plugin** | installed automatically | Fixtures plus a Gherkin step library for pytest-bdd / behave. | @@ -366,7 +440,7 @@ ignore synthetic input, and fall back silently when the driver is absent. ## Development ```bash -git clone https://github.com/Intergration-Automation-Testing/AutoControl.git +git clone https://github.com/Integration-Automation/AutoControlGUI.git cd AutoControl pip install -r dev_requirements.txt uv sync # or: reproducible install from the committed uv.lock @@ -394,6 +468,6 @@ API and a GUI surface. See [Third_Party_License.md](Third_Party_License.md) for the licenses of bundled and optional third-party components. -- **Homepage**: https://github.com/Intergration-Automation-Testing/AutoControl +- **Homepage**: https://github.com/Integration-Automation/AutoControlGUI - **PyPI**: https://pypi.org/project/je_auto_control/ - **Documentation**: https://autocontrol.readthedocs.io/en/latest/ diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 508c2bc5b..498f30dbb 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -5,8 +5,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](../LICENSE) [![Documentation](https://readthedocs.org/projects/autocontrol/badge/?version=latest)](https://autocontrol.readthedocs.io/en/latest/?badge=latest) -**AutoControl** 是一套跨平台的 Python GUI 自动化框架。它能驱动鼠标与键盘、在画面上找到目标 -(模板匹配、OCR、操作系统无障碍树,或视觉模型)、录制与回放操作流程,并以 JSON 动作文件执行—— +**AutoControl** 是一套开源、跨平台的 **computer-use 与 GUI 自动化框架**,面向 AI agent、Python 应用程序与自动化测试。它能驱动鼠标与键盘、通过模板匹配、OCR、操作系统无障碍树或视觉模型找到 UI 目标、录制与回放操作流程,并以 JSON 动作文件执行—— 支持 Windows、macOS、Linux(X11 与 Wayland)、BSD、Android 与 iOS。 每项能力都以三种形式提供:**Python API**、可在 JSON 文件/CLI/服务器使用的 **`AC_*` 动作命令**, @@ -27,8 +26,84 @@ - **依赖基线轻量。** REST 服务器、JSON Schema 校验、JWT、TOTP、WebSocket 帧、ACME 客户端、 USB/IP 协议与 Prometheus 指标全部以标准库实现;较重的依赖都是可选项。 + +## AI Agent 与 Computer Use + +**AutoControl 是开源、跨平台的 computer-use 与 GUI 自动化框架。** + +当 AI agent 需要操作真实桌面、原生应用程序、真实浏览器窗口、远程桌面或移动设备,而不只是操作浏览器 DOM/API 时,使用 AutoControl。 + +### 何时选择 AutoControl + +- 真实鼠标与键盘输入 +- 屏幕截图与坐标操作 +- OCR 与原生无障碍树 +- 图像/模板匹配 +- Vision-Language Model UI 定位 +- Anchor 与 self-healing locator +- 跨平台桌面/移动设备自动化 +- JSON 动作文件 +- MCP computer use + +建议的 agent loop: + +**observe → identify → act → wait → verify → recover** + +完整的 [AI Agent 指南](../AI.md) 提供 MCP 配置、工具选择、OpenAI 集成、安全性与 computer-use 关键词。 + +### AI-friendly MCP + +```bash +pip install je_auto_control +je_auto_control_mcp +``` + +MCP 提供完整的 `ac_*` 工具,以及 `click`、`type`、`screenshot`、`find_text`、`click_text`、`drag`、`focus_window`、`wait_image` 等短名称 alias。 + +只做检查: + +```bash +je_auto_control_mcp --read-only +``` + +只需要 canonical `ac_*` 名称: + +```bash +JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp +``` + +OpenAI agent 请使用 `export_openai_tools(only=[...])` 提供聚焦工具集,不要一次提供整个 AutoControl 命令目录。 + +### 项目识别 + +**项目:** AutoControl +**Repository:** `Integration-Automation/AutoControlGUI` +**Python package / PyPI:** `je_auto_control` +**MCP command:** `je_auto_control_mcp` + + --- + +## 如何选择自动化层 + +AutoControl 不需要取代所有自动化工具;应选择最符合界面的那一层: + +| 需求 | 适合工具 | +|---|---| +| 稳定的浏览器 DOM/API 自动化 | Playwright/Selenium | +| 简单的 Python 鼠标键盘脚本 | PyAutoGUI 或 AutoControl | +| 原生桌面应用程序自动化 | **AutoControl** | +| 无障碍树 GUI 自动化 | **AutoControl** | +| OCR GUI 自动化 | **AutoControl** | +| 截图/视觉模型 GUI 定位 | **AutoControl** | +| Self-healing 跨平台 GUI locator | **AutoControl** | +| AI agent 操作真实桌面 | **AutoControl + MCP** | +| 确定性的 JSON GUI workflow | **AutoControl** | + +AutoControl 的差异在于把 **真实电脑输入 + 语义/视觉定位 + self-healing + agent/MCP 集成** 放在同一个跨平台自动化接口中。 + + ## 安装 ```bash @@ -191,7 +266,7 @@ je_auto_control version | 接口 | 启动方式 | 说明 | |---|---|---| -| **MCP 服务器** | `je_auto_control_mcp`(stdio)或 `AC_start_mcp_http_server` | 678 个工具,供 Claude Desktop/Claude Code/自定义 tool loop 使用。除了以 `initialize` 握手的各版协议,也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | +| **MCP 服务器** | `je_auto_control_mcp`(stdio)或 `AC_start_mcp_http_server` | 完整的 `ac_*` 工具接口,供 Claude Desktop/Claude Code/自定义 tool loop 使用,并提供常用 GUI 操作的短名称 alias。除了以 `initialize` 握手的各版协议,也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、按 IP 限流与锁定、SQLite 审计 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 服务器** | `je_auto_control start-server` | 以换行分隔的 JSON 动作列表。默认绑定 `127.0.0.1`。 | | **pytest 插件** | 安装后自动生效 | 提供 fixture 与供 pytest-bdd/behave 使用的 Gherkin step library。 | @@ -332,7 +407,7 @@ Windows、macOS(pyobjc)与 X11(含 XWayland);纯 Wayland 会话的协 ## 开发 ```bash -git clone https://github.com/Intergration-Automation-Testing/AutoControl.git +git clone https://github.com/Integration-Automation/AutoControlGUI.git cd AutoControl pip install -r dev_requirements.txt uv sync # 或:以已提交的 uv.lock 做可重现安装 @@ -358,6 +433,6 @@ bandit -c pyproject.toml -r je_auto_control/ [MIT License](../LICENSE) © JE-Chen。 内含与可选第三方组件的许可请见 [Third_Party_License.md](../Third_Party_License.md)。 -- **主页**:https://github.com/Intergration-Automation-Testing/AutoControl +- **主页**:https://github.com/Integration-Automation/AutoControlGUI - **PyPI**:https://pypi.org/project/je_auto_control/ - **文档**:https://autocontrol.readthedocs.io/en/latest/ diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 404537ccd..92c72abca 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -5,9 +5,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](../LICENSE) [![Documentation](https://readthedocs.org/projects/autocontrol/badge/?version=latest)](https://autocontrol.readthedocs.io/en/latest/?badge=latest) -**AutoControl** 是一套跨平台的 Python GUI 自動化框架。它能驅動滑鼠與鍵盤、在畫面上找到目標 -(樣板比對、OCR、作業系統無障礙樹,或視覺模型)、錄製與重播操作流程,並以 JSON 動作檔執行—— -支援 Windows、macOS、Linux(X11 與 Wayland)、BSD、Android 與 iOS。 +**AutoControl** 是一套開源、跨平台的 **computer-use 與 GUI 自動化框架**,面向 AI agent、Python 應用程式與自動化測試。它能驅動滑鼠與鍵盤、透過樣板比對、OCR、作業系統無障礙樹或視覺模型找到 UI 目標、錄製與重播流程,並以 JSON 動作檔執行——支援 Windows、macOS、Linux(X11 與 Wayland)、BSD、Android 與 iOS。 每項能力都以三種形式提供:**Python API**、可在 JSON 檔/CLI/伺服器使用的 **`AC_*` 動作指令**, 以及 **GUI 分頁**。沒有任何功能只存在於 GUI。 @@ -27,8 +25,84 @@ - **相依基線輕薄。** REST 伺服器、JSON Schema 驗證、JWT、TOTP、WebSocket 框架、ACME 用戶端、 USB/IP 協定與 Prometheus 指標全部以標準庫實作;較重的相依都是選用。 + +## AI Agent 與 Computer Use + +**AutoControl 是開源、跨平台的 computer-use 與 GUI 自動化框架。** + +當 AI agent 需要操作真實桌面、原生應用程式、真正的瀏覽器視窗、遠端桌面或行動裝置,而不只是操作瀏覽器 DOM/API 時,使用 AutoControl。 + +### 何時選擇 AutoControl + +- 真實滑鼠與鍵盤輸入 +- 螢幕擷取與座標操作 +- OCR 與原生無障礙樹 +- 影像/樣板比對 +- Vision-Language Model UI 定位 +- Anchor 與 self-healing locator +- 跨平台桌面/行動裝置自動化 +- JSON 動作檔 +- MCP computer use + +建議的 agent loop: + +**observe → identify → act → wait → verify → recover** + +完整的 [AI Agent 指南](../AI.md) 提供 MCP 設定、工具選擇、OpenAI 整合、安全性與 computer-use 關鍵字。 + +### AI-friendly MCP + +```bash +pip install je_auto_control +je_auto_control_mcp +``` + +MCP 提供完整的 `ac_*` 工具,以及 `click`、`type`、`screenshot`、`find_text`、`click_text`、`drag`、`focus_window`、`wait_image` 等短名稱 alias。 + +只做檢查: + +```bash +je_auto_control_mcp --read-only +``` + +只需要 canonical `ac_*` 名稱: + +```bash +JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp +``` + +OpenAI agent 請使用 `export_openai_tools(only=[...])` 提供聚焦工具集,不要一次提供整個 AutoControl 命令目錄。 + +### 專案識別 + +**專案:** AutoControl +**Repository:** `Integration-Automation/AutoControlGUI` +**Python package / PyPI:** `je_auto_control` +**MCP command:** `je_auto_control_mcp` + + --- + +## 如何選擇自動化層 + +AutoControl 不需要取代所有自動化工具;應該選擇最符合介面的那一層: + +| 需求 | 適合工具 | +|---|---| +| 穩定的瀏覽器 DOM/API 自動化 | Playwright/Selenium | +| 簡單的 Python 滑鼠鍵盤腳本 | PyAutoGUI 或 AutoControl | +| 原生桌面應用程式自動化 | **AutoControl** | +| 無障礙樹 GUI 自動化 | **AutoControl** | +| OCR GUI 自動化 | **AutoControl** | +| 截圖/視覺模型 GUI 定位 | **AutoControl** | +| Self-healing 跨平台 GUI locator | **AutoControl** | +| AI agent 操作真實桌面 | **AutoControl + MCP** | +| 確定性的 JSON GUI workflow | **AutoControl** | + +AutoControl 的差異在於把 **真實電腦輸入 + 語意/視覺定位 + self-healing + agent/MCP 整合** 放在同一個跨平台自動化介面。 + + ## 安裝 ```bash @@ -191,7 +265,7 @@ je_auto_control version | 介面 | 啟動方式 | 說明 | |---|---|---| -| **MCP 伺服器** | `je_auto_control_mcp`(stdio)或 `AC_start_mcp_http_server` | 678 個工具,供 Claude Desktop/Claude Code/自訂 tool loop 使用。除了以 `initialize` 握手的各版協定,也支援無狀態的 MCP 2026-07-28。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | +| **MCP 伺服器** | `je_auto_control_mcp`(stdio)或 `AC_start_mcp_http_server` | 完整的 `ac_*` 工具介面,供 Claude Desktop/Claude Code/自訂 tool loop 使用,並提供常用 GUI 操作的短名稱 alias。除了以 `initialize` 握手的各版協定,也支援無狀態的 MCP 2026-07-28。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、逐 IP 限流與鎖定、SQLite 稽核 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 伺服器** | `je_auto_control start-server` | 以換行分隔的 JSON 動作清單。預設綁 `127.0.0.1`。 | | **pytest 外掛** | 安裝後自動生效 | 提供 fixture 與供 pytest-bdd/behave 使用的 Gherkin step library。 | @@ -333,7 +407,7 @@ Windows、macOS(pyobjc)與 X11(含 XWayland);純 Wayland session 的 ## 開發 ```bash -git clone https://github.com/Intergration-Automation-Testing/AutoControl.git +git clone https://github.com/Integration-Automation/AutoControlGUI.git cd AutoControl pip install -r dev_requirements.txt uv sync # 或:以已提交的 uv.lock 做可重現安裝 @@ -359,6 +433,6 @@ bandit -c pyproject.toml -r je_auto_control/ [MIT License](../LICENSE) © JE-Chen。 內含與選用第三方元件的授權請見 [Third_Party_License.md](../Third_Party_License.md)。 -- **首頁**:https://github.com/Intergration-Automation-Testing/AutoControl +- **首頁**:https://github.com/Integration-Automation/AutoControlGUI - **PyPI**:https://pypi.org/project/je_auto_control/ - **文件**:https://autocontrol.readthedocs.io/en/latest/ diff --git a/dev.toml b/dev.toml index 2bd513bc9..0b331cb40 100644 --- a/dev.toml +++ b/dev.toml @@ -51,9 +51,9 @@ je_auto_control_mcp = "je_auto_control.utils.mcp_server.__main__:main" je_auto_control = "je_auto_control.utils.pytest_plugin.plugin" [project.urls] -Homepage = "https://github.com/Intergration-Automation-Testing/AutoControl" +Homepage = "https://github.com/Integration-Automation/AutoControlGUI" Documentation = "https://autocontrol.readthedocs.io/en/latest/" -Code = "https://github.com/Intergration-Automation-Testing/AutoControl" +Code = "https://github.com/Integration-Automation/AutoControlGUI" [project.readme] file = "README.md" diff --git a/docs/source/Eng/doc/ai_agents/ai_agents_doc.rst b/docs/source/Eng/doc/ai_agents/ai_agents_doc.rst new file mode 100644 index 000000000..3d050b8f2 --- /dev/null +++ b/docs/source/Eng/doc/ai_agents/ai_agents_doc.rst @@ -0,0 +1,120 @@ +AI agents and computer use +=========================== + +AutoControl is an open-source, cross-platform computer-use and GUI automation framework for AI agents. + +Use it when an agent needs to control a real desktop GUI rather than only a browser DOM or API. + +When to use AutoControl +----------------------- + +AutoControl is a good fit for: + +* real mouse and keyboard input; +* screenshots and screen-coordinate interaction; +* OCR and accessibility-tree discovery; +* image/template matching; +* vision-language-model UI localization; +* anchor and self-healing locators; +* native desktop applications and real browsers; +* cross-platform desktop and mobile automation; +* deterministic JSON action files; +* MCP-based computer use. + +MCP for AI agents +----------------- + +Install and start the stdio MCP server: + +.. code-block:: bash + + pip install je_auto_control + je_auto_control_mcp + +The server exposes the canonical ac_* tools and short, model-friendly aliases for common actions: + +* click +* move_mouse +* scroll +* type +* press +* hotkey +* screenshot +* screen_size +* find_image +* find_text +* click_text +* drag +* list_windows +* focus_window +* wait_image +* wait_pixel + +Disable aliases when a client needs only canonical tools: + +.. code-block:: bash + + JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp + +Use read-only mode for inspection-only clients: + +.. code-block:: bash + + je_auto_control_mcp --read-only + +Recommended agent loop +---------------------- + +#. Observe the current screen. +#. Identify the target with accessibility, OCR, image matching, or VLM. +#. Perform the smallest required action. +#. Wait for the interface to settle. +#. Verify the expected result. +#. Retry with another locator strategy if the UI changed. + +OpenAI tool selection +--------------------- + +OpenAI Chat Completions has a provider-side tool-count limit. Do not pass the complete AutoControl catalogue to an OpenAI agent. Export a focused allow-list: + +.. code-block:: python + + from je_auto_control.utils.tool_use_schema import export_openai_tools + + tools = export_openai_tools(only=[ + "AC_screenshot", + "AC_click_mouse", + "AC_write", + "AC_hotkey", + "AC_click_text", + ]) + +Focused toolsets are also safer: keep shell, process execution, package loading, and recursive agent commands out of the list unless explicitly required. + +Examples +-------- + +For the closed-loop Python agent see the repository example at examples/05_agent_loop.py. + +A stdio MCP client can launch AutoControl with: + +.. code-block:: json + + { + "mcpServers": { + "autocontrol": { + "command": "je_auto_control_mcp" + } + } + } + +Security +-------- + +An AI-controlled GUI process has control over the host machine. + +* Keep servers on loopback unless remote access is intentional. +* Use --read-only for inspection-only clients. +* Prefer explicit tool allow-lists. +* Do not expose shell/process/package-loading tools to untrusted models. +* Use authentication, audit, rate limiting, and confirmation controls for service deployments. diff --git a/docs/source/Eng/doc/new_features/v2_features_doc.rst b/docs/source/Eng/doc/new_features/v2_features_doc.rst index 8a211b557..ba7f0eb99 100644 --- a/docs/source/Eng/doc/new_features/v2_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v2_features_doc.rst @@ -399,10 +399,10 @@ Generic agent loop (JSON + MCP) language and the MCP tool registry. Parameters: * ``goal`` — natural-language objective. -* ``backend`` — ``"anthropic"`` (uses ``export_anthropic_tools()`` - with tool-use messages; each screenshot is fitted into the model's image - tier and the ``x`` / ``y`` of a tool call mapped back to the screen) or ``"openai"`` (uses ``export_openai_tools()`` - with Chat Completions function calling). +* ``backend`` — ``"anthropic"`` or ``"openai"``. ``AC_run_agent`` uses a focused, low-risk + computer-use allow-list by default instead of exposing the complete ``AC_*`` catalogue. + Applications that need a custom set should construct the backend with + ``export_anthropic_tools(only=[...])`` or ``export_openai_tools(only=[...])``. * ``max_steps`` (default 25) and ``wall_seconds`` (default 300.0). * ``model`` / ``max_tokens`` — backend-specific overrides. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index e3f679e68..e826e9bb4 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -20,6 +20,8 @@ Comprehensive guides for all AutoControl features. doc/scheduler/scheduler_doc doc/socket_driver/socket_driver_doc doc/mcp_server/mcp_server_doc + doc/ai_agents/ai_agents_doc + doc/ai_agents/ai_agents_doc doc/critical_exit/critical_exit_doc doc/cli/cli_doc doc/create_project/create_project_doc diff --git a/docs/source/Zh/doc/ai_agents/ai_agents_doc.rst b/docs/source/Zh/doc/ai_agents/ai_agents_doc.rst new file mode 100644 index 000000000..484023335 --- /dev/null +++ b/docs/source/Zh/doc/ai_agents/ai_agents_doc.rst @@ -0,0 +1,116 @@ +AI Agent 與 Computer Use +=========================== + +AutoControl 是開源、跨平台的 **computer-use 與 GUI 自動化框架**,可以讓 AI agent 操作真實桌面,而不只依賴瀏覽器 DOM 或 API。 + +何時使用 AutoControl +-------------------- + +適合以下情境: + +* 真實滑鼠與鍵盤輸入; +* 螢幕擷取與座標操作; +* OCR 與作業系統無障礙樹; +* 影像/樣板比對; +* Vision-Language Model UI 定位; +* Anchor 與 self-healing locator; +* 原生桌面應用程式與真實瀏覽器; +* 跨平台桌面與行動裝置自動化; +* JSON 動作檔; +* 透過 MCP 讓 AI agent 使用 computer use。 + +MCP for AI agents +----------------- + +安裝並啟動 stdio MCP server: + +.. code-block:: bash + + pip install je_auto_control + je_auto_control_mcp + +MCP 同時提供完整的 ac_* 工具,以及給模型使用的短名稱 alias: + +* click +* move_mouse +* scroll +* type +* press +* hotkey +* screenshot +* screen_size +* find_image +* find_text +* click_text +* drag +* list_windows +* focus_window +* wait_image +* wait_pixel + +若 client 只需要 canonical ac_* 工具: + +.. code-block:: bash + + JE_AUTOCONTROL_MCP_ALIASES=0 je_auto_control_mcp + +只做檢查、不允許修改的 client: + +.. code-block:: bash + + je_auto_control_mcp --read-only + +建議的 Agent Loop +----------------- + +#. 觀察目前畫面。 +#. 優先用無障礙樹、OCR、影像比對或 VLM 找目標。 +#. 執行最小必要操作。 +#. 等待介面穩定。 +#. 驗證預期結果。 +#. UI 改變時切換另一種 locator 策略重新嘗試。 + +OpenAI 工具選擇 +--------------- + +OpenAI Chat Completions 有 provider 的工具數量限制,因此不要把整個 AutoControl 命令目錄一次交給 OpenAI agent。請針對任務建立 allow-list: + +.. code-block:: python + + from je_auto_control.utils.tool_use_schema import export_openai_tools + + tools = export_openai_tools(only=[ + "AC_screenshot", + "AC_click_mouse", + "AC_write", + "AC_hotkey", + "AC_click_text", + ]) + +聚焦工具集也比較安全:除非真的需要,否則不要提供 shell、process execution、package loading 或遞迴 agent 工具。 + +MCP client 範例 +--------------- + +stdio MCP client 可以直接啟動 AutoControl: + +.. code-block:: json + + { + "mcpServers": { + "autocontrol": { + "command": "je_auto_control_mcp" + } + } + } + +安全性 +------ + +AI 控制的 GUI process 具有主機操作權限。 + +* 非必要不要把 server 綁到公開網路。 +* 檢查型 client 使用 --read-only。 +* AI agent 優先使用明確的工具 allow-list。 +* 不要把 shell/process/package-loading 工具暴露給不受信任的模型。 +* 作為服務部署時使用 authentication、audit、rate limit 與 confirmation 控制。 diff --git a/docs/source/Zh/doc/new_features/v2_features_doc.rst b/docs/source/Zh/doc/new_features/v2_features_doc.rst index 81e5045ed..d5b457d7f 100644 --- a/docs/source/Zh/doc/new_features/v2_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v2_features_doc.rst @@ -369,9 +369,9 @@ helper(``je_auto_control.gui.flow_editor.layout_steps``)可單元 (規劃 → 執行 → 驗證 → 重試)開放給 JSON action 與 MCP。參數: * ``goal`` — 自然語言目標。 -* ``backend`` — ``"anthropic"``(透過 ``export_anthropic_tools()`` - 以 tool-use messages 驅動;每張截圖先縮到模型的影像層級內,工具呼叫的 ``x`` / ``y`` 再換算回螢幕)或 ``"openai"``(``export_openai_tools()`` - + Chat Completions function calling)。 +* ``backend`` — ``"anthropic"`` 或 ``"openai"``。``AC_run_agent`` 預設只提供聚焦、低風險的 + computer-use allow-list,不再把完整的 ``AC_*`` 命令目錄交給模型。需要自訂工具集時, + 可直接用 ``export_anthropic_tools(only=[...])`` 或 ``export_openai_tools(only=[...])`` 建立 backend。 * ``max_steps``(預設 25)、``wall_seconds``(預設 300.0)。 * ``model`` / ``max_tokens`` — backend 專屬覆寫。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index b857826f4..eb1f7a49f 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -20,6 +20,8 @@ AutoControl 所有功能的完整使用指南。 doc/scheduler/scheduler_doc doc/socket_driver/socket_driver_doc doc/mcp_server/mcp_server_doc + doc/ai_agents/ai_agents_doc + doc/ai_agents/ai_agents_doc doc/critical_exit/critical_exit_doc doc/cli/cli_doc doc/create_project/create_project_doc diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 344fd9335..80a34f74b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -103,3 +103,14 @@ WebRunner's native commands (its U-20261001-28 and -29), `WR_ac_fill_native_file - **Docs**: `architecture.md` (§2 `scripts/` row, §3 PyPI packages row), `architecture_explore.md` (§5.6 `scripts/dev_release.py` row, §7 `dev.yml` row), `CLAUDE.md` › Key Conventions, `CHANGELOG.md` (Changed). No README or installation page mentions the dev package, so none changed. - **Files**: `.github/workflows/dev.yml`, `scripts/dev_release.py`, `dev.toml`, `test/unit_test/headless/{test_dev_release,test_dev_toml_parity}.py`, `architecture.md`, `architecture_explore.md`, `CLAUDE.md`, `CHANGELOG.md`. - **Open items**: the points under **Not running yet**. `Progress.md` is not part of this change, so it does not list them. + + +## U-20261006-01 · 2026-10-06 · Make AutoControl discoverable and safer for AI agents · #feature #ai #mcp #docs + +- **Positioning**: the English and Chinese READMEs now describe AutoControl as a cross-platform computer-use and GUI automation framework for AI agents, with explicit guidance on when an agent should choose it. +- **AI documentation**: added `AI.md` plus bilingual Sphinx pages covering MCP setup, model-friendly aliases, recommended observe → identify → act → wait → verify → recover loops, OpenAI tool selection, security, and project identity. +- **MCP discoverability**: documented the existing short aliases (`click`, `type`, `screenshot`, `find_text`, `click_text`, `drag`, `focus_window`, `wait_image`) and read-only / canonical-only modes. +- **Agent safety**: `AC_run_agent` now gives Anthropic and OpenAI backends a focused computer-use allow-list by default. System-command, package-loading, recursive-agent and other high-authority commands are not offered by default. Custom agents can still use `export_*_tools(only=[...])`. +- **MCP registry identity**: corrected the registry server name and repository URL to the current `Integration-Automation/AutoControlGUI` organization, and aligned `pyproject.toml` URLs. +- **Regression coverage**: added a headless test ensuring the OpenAI agent path receives a focused tool set and remains within the provider tool-count limit. +- **Files**: `AI.md`, bilingual AI-agent docs, README translations, `pyproject.toml`, MCP registry manifest builder, `action_executor.py`, agent wiring test, `CHANGELOG.md`, `Progress.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 08d1cd29c..1e3a05eec 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,7 +58,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| -| U-20261001-09 | 2026-10-01 | CI publishes je_auto_control_dev from the dev branch; dev.toml says what pyproject.toml says | #release #ci #X-13 | [2026-10](2026-10.md) | +| U-20261006-01 | 2026-10-06 | Make AutoControl discoverable and safer for AI agents | #feature #ai #mcp #docs | [2026-10](2026-10.md) |\n| U-20261001-09 | 2026-10-01 | CI publishes je_auto_control_dev from the dev branch; dev.toml says what pyproject.toml says | #release #ci #X-13 | [2026-10](2026-10.md) | | U-20261001-08 | 2026-10-01 | Package gate in front of AC_add_package_to_executor | #security #X-12 | [2026-10](2026-10.md) | | U-20261001-07 | 2026-10-01 | write_secret / AC_write_secret: type a password without logging, recording or returning it | #done #keyboard #security #webrunner | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Pin the AC commands WebRunner's native WR_ac_* send; record the missing secret typing | #contract #webrunner | [2026-10](2026-10.md) | diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 89b25c14e..14629d369 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -689,6 +689,30 @@ def _presence_clear() -> Dict[str, Any]: return {"cleared": True} +_DEFAULT_AGENT_TOOLSET = [ + "AC_screenshot", + "AC_screen_size", + "AC_set_mouse_position", + "AC_get_mouse_position", + "AC_click_mouse", + "AC_mouse_scroll", + "AC_drag", + "AC_write", + "AC_type_keyboard", + "AC_hotkey", + "AC_press_key", + "AC_locate_image_center", + "AC_click_text", + "AC_wait_text", + "AC_wait_image", + "AC_a11y_find", + "AC_a11y_click", + "AC_list_windows", + "AC_focus_window", + "AC_assert_text", +] + + def _run_agent(goal: str, backend: str = "anthropic", max_steps: int = 25, @@ -698,8 +722,11 @@ def _run_agent(goal: str, """Executor adapter: drive the closed-loop ``AgentLoop`` against ``goal``. ``backend`` selects between the production backends (Anthropic / - OpenAI). The Anthropic computer-use raw path remains available - via :func:`_computer_use` / ``AC_computer_use``. + OpenAI). Both receive a small, safe computer-use toolset by default; + the raw Anthropic computer-use path remains available via + :func:`_computer_use` / ``AC_computer_use``. Applications that need a + larger custom toolset should construct the agent backend directly with + ``export_*_tools(only=[...])``. """ from je_auto_control.utils.agent import AgentBudget, AgentLoop from je_auto_control.utils.agent.agent_loop import AgentBackend @@ -712,14 +739,14 @@ def _run_agent(goal: str, name = (backend or "anthropic").strip().lower() backend_obj: AgentBackend if name == "anthropic": - tools = export_anthropic_tools() + tools = export_anthropic_tools(only=_DEFAULT_AGENT_TOOLSET) backend_obj = AnthropicAgentBackend( tools=tools, model=model or "claude-opus-4-7", max_tokens=int(max_tokens), ) elif name == "openai": - tools = export_openai_tools() + tools = export_openai_tools(only=_DEFAULT_AGENT_TOOLSET) # OpenAIAgentBackend does not accept max_tokens (Anthropic-only). backend_obj = OpenAIAgentBackend( tools=tools, diff --git a/je_auto_control/utils/mcp_registry/registry.py b/je_auto_control/utils/mcp_registry/registry.py index 632edb5ae..679bab237 100644 --- a/je_auto_control/utils/mcp_registry/registry.py +++ b/je_auto_control/utils/mcp_registry/registry.py @@ -16,13 +16,13 @@ _SCHEMA_URL = ("https://static.modelcontextprotocol.io/schemas/" "2025-09-29/server.schema.json") -_SERVER_NAME = "io.github.intergration-automation-testing/autocontrol" -_REPO_URL = "https://github.com/Intergration-Automation-Testing/AutoControl" +_SERVER_NAME = "io.github.Integration-Automation/autocontrolgui" +_REPO_URL = "https://github.com/Integration-Automation/AutoControlGUI" _PYPI_NAME = "je_auto_control" _DEFAULT_VERSION = "0.0.189" # The registry schema caps description at 100 characters; the old 161 would # have been rejected on publish. -_DESCRIPTION = "Cross-platform GUI automation: mouse, keyboard, image/OCR and accessibility as MCP tools" +_DESCRIPTION = "Cross-platform computer-use and GUI automation for AI agents via MCP" _MAX_DESCRIPTION = 100 # The one _meta key the registry schema defines for publisher data. _META_KEY = "io.modelcontextprotocol.registry/publisher-provided" diff --git a/pyproject.toml b/pyproject.toml index aaf9bf06e..20a511191 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -74,9 +74,9 @@ je_auto_control_mcp = "je_auto_control.utils.mcp_server.__main__:main" je_auto_control = "je_auto_control.utils.pytest_plugin.plugin" [project.urls] -Homepage = "https://github.com/Intergration-Automation-Testing/AutoControl" +Homepage = "https://github.com/Integration-Automation/AutoControlGUI" Documentation = "https://autocontrol.readthedocs.io/en/latest/" -Code = "https://github.com/Intergration-Automation-Testing/AutoControl" +Code = "https://github.com/Integration-Automation/AutoControlGUI" [project.readme] file = "README.md" diff --git a/test/unit_test/headless/test_agent_executor_mcp_wiring.py b/test/unit_test/headless/test_agent_executor_mcp_wiring.py index 132426e14..e9c991e86 100644 --- a/test/unit_test/headless/test_agent_executor_mcp_wiring.py +++ b/test/unit_test/headless/test_agent_executor_mcp_wiring.py @@ -80,3 +80,26 @@ def test_unknown_backend_raises(): import pytest with pytest.raises(ValueError, match="unknown agent backend"): _run_agent(goal="x", backend="bogus") + + +def test_openai_agent_uses_focused_toolset(monkeypatch): + captured = {} + + class StubBackend(FakeAgentBackend): + def __init__(self, *, tools, **kwargs): + captured["tools"] = tools + super().__init__([{"stop": True, "message": "focused"}]) + + import je_auto_control.utils.agent.backends as backends_pkg + monkeypatch.setattr(backends_pkg, "OpenAIAgentBackend", StubBackend) + from je_auto_control.utils.executor.action_executor import _run_agent + + result = _run_agent(goal="probe", backend="openai", max_steps=1, wall_seconds=5.0) + + names = {item["function"]["name"] for item in captured["tools"]} + assert result["succeeded"] is True + assert "AC_screenshot" in names + assert "AC_click_mouse" in names + assert "AC_shell_command" not in names + assert "AC_execute_process" not in names + assert len(names) <= 128