macOS GUI Automation Skill#

Capabilities#

• Screen Reading: Capture screenshots and OCR text
• Mouse Control: Click, double-click, right-click, move, drag
• Keyboard Input: Type text, press keys, shortcuts
• Window Management: List windows, focus, resize, close
• App Control: Launch, quit, bring to front

Tools Available#

cliclick (Mouse/Keyboard)#

bash

# Click at coordinates
cliclick c:x,y

# Double click
cliclick dc:x,y

# Right click
cliclick rc:x,y

# Move mouse
cliclick m:x,y

# Drag from to
cliclick dr:x1,y1:x2,y2

# Type text
cliclick t:"text"

# Press key (Enter, Tab, etc.)
cliclick kp:enter

screencapture + tesseract (Screen Reading)#

bash

# Capture region to file
screencapture -R x,y,w,h /tmp/screen.png

# Capture full screen
screencapture /tmp/screen.png

# OCR from image
tesseract /tmp/screen.png stdout

# OCR with Chinese support
tesseract /tmp/screen.png stdout -l chi_sim+eng

osascript (AppleScript - Window/App Control)#

bash

# List all windows
osascript -e 'tell application "System Events" to get name of every process'

# Get window position/size
osascript -e 'tell application "Finder" to get bounds of window of front window'

# Click menu item
osascript -e 'tell application "System Events" to click menu item "Save" of menu "File" of process "TextEdit"'

Usage Patterns#

Read Screen Text#

bash

# 1. Capture screen
screencapture -R 100,100,800,600 /tmp/region.png

# 2. OCR
tesseract /tmp/region.png stdout

Click Button at Position#

bash

cliclick c:500,300

Type in Field#

bash

# Click field first
cliclick c:400,200
# Then type
cliclick t:"hello world"
cliclick kp:enter

Find and Click (OCR + Click)#

bash

# 1. Capture and OCR
screencapture /tmp/screen.png
text=$(tesseract /tmp/screen.png stdout)

# 2. Parse coordinates from OCR result or use image recognition

# 3. Click
cliclick c:x,y

Limitations#

• Coordinates are absolute (screen resolution dependent)
• No built-in image recognition (need to add OpenCV/sikuli for that)
• OCR accuracy depends on screen DPI and font
• Some apps may not be scriptable via AppleScript

Security Notes#

• Requires Accessibility permissions in System Settings
• Run tccutil reset Accessibility if permissions issues
• Some apps (browsers, secure apps) may block automation