Telegram UI

Auto_Use/macOS_use/agent/main_driver/system_prompt.md

@@ -77,7 +77,9 @@ Each step includes:
 1. decision: Decision made based on images.
 2. next_goal: Forward plan — the immediate step plus the next few anticipated steps.
 3. memory: Key information stored.
-4. action: Action performed.
+4. action: Action performed that step.
+*The most recent step also keeps its `thinking` and `eval`; older steps are trimmed to the fields above to save context.
+*Each step's tool result follows it as a `<tool_response>` user turn — this is how you see what your action produced (e.g. click outcome with element_name, shell output). Web results are summarized there; the raw data is saved to <scratchpad>.
 </agent_history>
 <Tool_Capability>
 *Use tools only inside the action list.*
@@ -145,8 +147,9 @@ Each step includes:
 2. Only write after visual confirmation — never assume success.
 3. Write immediately when something is confirmed. If multiple facts are confirmed in one step, emit one separate scratchpad action per fact.
 4. Use for: major task completions, metrics/numbers/final answers, important web findings, exact file save paths + filenames.
-5. Format: {"type": "scratchpad", "value": "one-line_verified_note"}
-6. Examples:
+5.Avoid writing repetitive information.
+6. Format: {"type": "scratchpad", "value": "one-line_verified_note"}
+7. Examples:
   1. {"type": "scratchpad", "value": "Done: Email sent to abc@gmail.com with flight details + attachments"}
   2. {"type": "scratchpad", "value": "Saved abc.pdf to ~/Documents/testing/abc.pdf"}
   3. {"type": "scratchpad", "value": "Key metric: Disney+ revenue (Q3 2025) = 2.1B $"}
@@ -165,7 +168,7 @@ Each step includes:
 <reasoning_rules>
 *You must reason explicitly and systematically at every step in your thinking block. Work through the rules below as five labeled stages — OBSERVE → VERIFY → PROGRESS → PLAN → PREDICT — to successfully achieve the objective:*
 1. Reason about <agent_history> to track progress and context toward <user_request>.
-2. Analyse the most recent "memory", "next_goal", and "action" in <agent_history> and clearly state what you previously planned and achieved (the "next_goal" lays out the immediate step plus the next 2-3 anticipated steps).
+2. Analyse the most recent "memory", "next_goal", "action" and its "<tool_response>" in <agent_history> and clearly state what you previously planned and achieved (the "next_goal" lays out the immediate step plus the next 2-3 anticipated steps).
 3. Analyse all the most relevant <agent_history>, <scratchpad>, <Tool_response>, <element_tree>, <todo_list>, <browser_guidlines> and the screenshot to understand your current state.
 4. Judge success/failure of the last action using <os_vision> as primary ground truth (not <last_response>). Feed your conclusion into "eval".
   1. Example: you might have `"action": [{"input": {"74": "abc@gmail.com"}}]` with a success response in <last_response>, even though inputting text actually failed. If the expected change is missing on screen, mark "eval" as FAIL and plan a recovery.

Auto_Use/macOS_use/controller/cli/service.py

@@ -175,23 +175,35 @@ def shell(self, command: str, input_text: str = None) -> dict:
                 output += result["stdout"]
             if result.get("stderr"):
                 output += result["stderr"]
-
+
+            returncode = result.get("returncode")
+            succeeded = result.get("success")
+
             response = {
-                "status": "success" if result.get("success") else "error",
+                "status": "success" if succeeded else "error",
                 "action": "shell",
                 "cwd": self.sandbox.get_cwd(),
                 "command": command
             }
-
-            # Only include output if there's actual content
-            if output.strip():
-                response["output"] = output.strip()
-
+
+            # Always surface the exit code so the agent can tell a real error
+            # from an empty result (e.g. grep exits 1 on no match).
+            if returncode is not None:
+                response["exit_code"] = returncode
+
+            # Show exactly what the command returned (verbatim stdout + stderr).
+            # When the terminal returns absolutely nothing, show a clear marker
+            # instead of a blank string — the status/exit_code already say whether
+            # it succeeded, so this just makes "no output" unambiguous (like a
+            # real terminal / Claude Code).
+            clean_output = output.strip()
+            response["output"] = clean_output if clean_output else "(no output)"
+
             # Only include error if there's actual content
             error = result.get("error", "")
             if error:
                 response["error"] = error
-            
+
             return response
 
         except Exception as e:

Auto_Use/macOS_use/controller/service.py

@@ -592,23 +592,29 @@ def scroll(self, index, direction):
             # Warp cursor to scroll target
             self._warp_cursor_only(center_x, center_y)
 
+            # macOS scrolls in LINE units — pyautogui posts one scroll event per unit,
+            # so this is a small, gentle nudge. This is NOT the Windows "120 == one wheel
+            # notch" convention: a value like 120 here becomes ~120 line events and macOS
+            # momentum scrolling then flings the view straight to the end. Keep it small.
+            # ~3 steps x 5 lines = a modest scroll; tune scroll_clicks to taste.
             scroll_amount = 3
+            scroll_clicks = 5
 
             if direction.lower() == "up":
                 for _ in range(scroll_amount):
-                    pyautogui.scroll(120, x=center_x, y=center_y)
+                    pyautogui.scroll(scroll_clicks, x=center_x, y=center_y)
                     time.sleep(0.05)
             elif direction.lower() == "down":
                 for _ in range(scroll_amount):
-                    pyautogui.scroll(-120, x=center_x, y=center_y)
+                    pyautogui.scroll(-scroll_clicks, x=center_x, y=center_y)
                     time.sleep(0.05)
             elif direction.lower() == "left":
                 for _ in range(scroll_amount):
-                    pyautogui.hscroll(-120, x=center_x, y=center_y)
+                    pyautogui.hscroll(-scroll_clicks, x=center_x, y=center_y)
                     time.sleep(0.05)
             elif direction.lower() == "right":
                 for _ in range(scroll_amount):
-                    pyautogui.hscroll(120, x=center_x, y=center_y)
+                    pyautogui.hscroll(scroll_clicks, x=center_x, y=center_y)
                     time.sleep(0.05)
             else:
                 return {"status": "error", "action": "scroll", "index": index,

Auto_Use/macOS_use/controller/tool/shell.py

@@ -92,21 +92,34 @@ def run(self, command: str, input_text: str = None) -> dict:
                 output += result["stdout"]
             if result.get("stderr"):
                 output += result["stderr"]
-
+
+            returncode = result.get("returncode")
+            succeeded = result.get("success")
+
             response = {
-                "status": "success" if result.get("success") else "failed",
+                "status": "success" if succeeded else "failed",
                 "action": "shell",
                 "agent_location": agent_location,
                 "shell": command,
             }
-
-            if output.strip():
-                response["output"] = output.strip()
-
+
+            # Always surface the exit code so the agent can tell a real error
+            # from an empty result (e.g. grep exits 1 on no match).
+            if returncode is not None:
+                response["exit_code"] = returncode
+
+            # Show exactly what the command returned (verbatim stdout + stderr).
+            # When the terminal returns absolutely nothing, show a clear marker
+            # instead of a blank string — the status/exit_code already say whether
+            # it succeeded, so this just makes "no output" unambiguous (like a
+            # real terminal / Claude Code).
+            clean_output = output.strip()
+            response["output"] = clean_output if clean_output else "(no output)"
+
             error = result.get("error", "")
             if error:
                 response["error"] = error
-            
+
             return response
 
         except Exception as e: