Skip to content

2026-05.md

Latest commit

 

History

History
57 lines (46 loc) · 6.14 KB

2026-05.md

File metadata and controls

57 lines (46 loc) · 6.14 KB
Raw

2026-05 update log

Index and query commands: README.md. New entries go at the end.

U-20260525-01 · 2026-05-25 · Self-healing locators, Wayland/Android/iOS, agent loop · #release #locators #platforms #agent

Twenty-seven additions covering smarter locators, deeper IDE / ops tooling, four new platforms (Wayland, Wayland-libei, Android widget-tree, iOS), screenshot PII redaction, and a generic plan-execute-verify agent loop. Each ships with a headless API, an AC_* executor command, an ac_* MCP tool, and (where it makes sense) a Qt GUI tab. Full reference page: docs/source/Eng/doc/new_features/v2_features_doc.rst.

Locator + selector intelligence

  • Self-healing locator — image_template → VLM fallback with a JSON-lines audit log (AC_self_heal_locate / _click).
  • Anchor-based locator — find element B by spatial relation (above, below, left_of, right_of, near) to anchor A; anchor and target can use different backends (image / OCR / VLM / a11y).
  • OCR with structured output — cluster raw OCR matches into rows, tables, and label:value form fields (AC_ocr_read_structure).
  • Smart waits — wait_until_screen_stable, wait_until_pixel_changes, wait_until_region_idle: frame-diff replacements for time.sleep.
  • A/B locator framework — race N strategies for the same target; recommend the historically best one from a persisted ledger.

Operations + observability

  • LLM cost telemetry — per-call token + USD log with day / model / provider rollup (record_llm_call, summarise_llm_costs).
  • Trace replay UI — scrubbable timeline over the existing time-travel recordings with per-step action list.
  • Failure → ticket automation — fan a failure report out to Jira / Linear / GitHub Issues when a scheduled / triggered / REST run fails.
  • Container CI templates — GitHub Actions + GitLab CI workflows that build the image, run the headless pytest suite under Xvfb, and smoke-test the REST entrypoint; XFCE+x11vnc Dockerfile variant for flows that need a real WM.
  • Cross-host DAG orchestrator — parallel execution with skip-on-failure cascade across local + admin-console-registered hosts (run_dag, AC_run_dag).
  • Multi-viewer presence — roster + controller/observer roles for the remote desktop, with a thread-safe Python PresenceRegistry independent of aiortc.

Agent + integrations

  • Computer-use high-level API — run_computer_use(goal, ...) wraps ComputerUseAgentBackend + AgentLoop; auto-detects display size; bounded by max_steps / wall_seconds.
  • Generic agent loop JSON + MCP — AC_run_agent / ac_run_agent expose the closed-loop AgentLoop (plan → act → verify → retry) with pluggable Anthropic / OpenAI backends; the Anthropic-only Computer-Use raw path remains via AC_computer_use.
  • WebRunner convenience commands — web_open / web_quit / web_screenshot / web_current_url on top of the existing je_web_runner bridge; same surface exposed as AC_web_* and ac_web_*.
  • Chat-ops bot — transport-agnostic CommandRouter + polling Slack adapter. Built-in commands: /help, /scripts, /run, /screenshot, /status. RBAC via required_role.

Privacy + safety

  • Screenshot PII redaction — RedactionEngine with built-in detectors for email / credit card / SSN / phone (regex against caller-supplied OCR tokens) plus accessibility-tree secure-text-field detection. Forced regions for sticky overlays. Env-var-driven default policy JE_AUTOCONTROL_REDACTION=off|moderate|strict. Wired through AC_redact_screenshot + ac_redact_screenshot.

Platform coverage

  • Wayland CLI backend — wtype / ydotool / grim with XDG_SESSION_TYPE auto-detect and X11 (XWayland) fallback; override via JE_AUTOCONTROL_LINUX_DISPLAY_SERVER=x11|wayland|auto.
  • Wayland libei native — ctypes binding to libei.so.* for microsecond-latency input; opt-in via JE_AUTOCONTROL_WAYLAND_INPUT_BACKEND=libei|cli|auto. Defaults to libei when loadable.
  • macOS Accessibility deep-dive — recursive dump_accessibility_tree() plus a polling AccessibilityRecorder for focus / bounds events.
  • Android — adb shell primitives — AC_android_tap/swipe/key/text/screenshot route through adb for any phone over USB / Wi-Fi adb. No daemon required.
  • Android — uiautomator2 widget tree — AC_android_find_element/click_element/dump_hierarchy add selector-based widget lookup (text / resource_id / description / class_name) and live XML hierarchy dump on top of the adb path.
  • iOS — XCUITest via WebDriverAgent — new je_auto_control.ios.* namespace: tap, swipe, long_press, type_text, press_key, screenshot, screen_size, find_element / click_element (XCUITest selectors: name, class_name, predicate), dump_source. Seven new AC_ios_* executor commands and matching ac_ios_* MCP tools. facebook-wda is an optional pip dep; loads lazily so non-Mac hosts still import the package.

Developer experience

  • autocontrol-lsp completion — the language server now tracks didOpen / didChange / didClose, publishes diagnostics for invalid JSON and unknown AC_* commands, and provides signature help generated from the live executor table.
  • .pyi stub generator — python -m je_auto_control.utils.stubs.generator je_auto_control/actions.pyi emits an IDE-facing stub so every AC_* command autocompletes with parameter hints.
  • VS Code extension — bundled extension now ships AutoControl: Run / Screenshot / Preview commands that hit the local REST API.
  • Browser extension recorder — Manifest V3 extension under browser-extension/: capture clicks, typing, navigation, form submissions in a tab and export them as AC_web_* / WR_* JSON.
  • pytest plugin + Gherkin BDD — pytest11 entry point auto-loads; @pytest.mark.autocontrol arms screenshot-on-failure; bdd_steps.register_pytest_bdd_steps(pytest_bdd) wires Given/When/Then onto every AC_* verb.
  • Visual flow editor — node-based view that round-trips to the same JSON action format the list-based Script Builder uses.