Windows Desktop Automation with winguictl#

⚠️ Security Notice#

This skill directly controls Windows desktop through mouse/keyboard simulation. Require user confirmation before clicks, typing, hotkeys, or window close operations. Use --dry-run to preview. Close sensitive apps before use.

Script Path#

powershell

python <SKILL_DIR>\scripts\winguictl.py <command> [options]

Commands#

CommandPurposeDocumentationwindowList/focus/minimize/maximize/close windowswindow.mdsnapshotGet HWND/UIA/OCR structuresnapshot.mdfindFind elements by text/UIA/OCR/imagefind.mdactionClick/drag/type/scroll/hotkeyaction.mdcontrolWin32 control operationscontrol.mduia-controlUIA element operationscontrol.mdscreenshotCapture window screenshotsscreenshot.mdwaitWait for conditionswait.mdclipboardCopy files/text, get textclipboard.md

Parameter Types#

ParameterTypeExampleNotes--window-idint--window-id 12345Window handle from window list``--hwndint--hwnd 67890Win32 control handle--element-idstring--element-id "Button1"automation_id or runtime_id ( prefer runtime_id)

Core Workflow#

powershell

# 1. Identify window
python scripts\winguictl.py window list

# 2. Focus window
python scripts\winguictl.py window --window-id <id> focus

# 3. Get snapshot
python scripts\winguictl.py snapshot --window-id <id> uia

# 4. Find & interact
python scripts\winguictl.py find --window-id <id> uia --text "Submit"
python scripts\winguictl.py uia-control --window-id <id> --element-id <elem_id> click

# 5. Verify (re-snapshot)
python scripts\winguictl.py snapshot --window-id <id> uia

Decision Guide#

Locator Priority#

PriorityMethodCommandReliability1HWND (Win32)control --hwnd <hwnd> clickHighest2runtime_id (UIA)uia-control --element-id <id> clickHigh3automation_id (UIA)uia-control --element-id <id> clickMedium4Image matchingaction click-image --image-path <path>Medium5Coordinatesaction click --relative-x/yLowest

Click Method Selection#

CommandMechanismBest Foraction click --element-idMouse simulation at centerWeChat, custom controlsuia-control clickUIA InvokePatternStandard Windows controls, WinUI3

Recommendation: Try uia-control click first. If no effect, use action click --element-id.

Common Scenarios#

ScenarioCommandWin32 control with hwndcontrol --hwnd <hwnd> clickUIA element with runtime_iduia-control --element-id "42-3155764" clickText-based elementfind ocr "text" → action click --element-idQt applicationsAdd --skip-actions --skip-state for faster UIA

Key Rules#

• Re-snapshot after actions: UI state changes after clicks/typing
• Use --dry-run: Preview operations before execution
• Prefer runtime_id: More reliable than automation_id
• Report window_id: Always include exact window identifier in results
• Coordinate system: relative_rect = window-relative; absolute_rect = screen-absolute

Application Guides#

• WeChat Automation — WeChat 4.1.6+ messaging, contacts, files, calls, Moments

References#

• Dependencies — pywinauto, pywin32, Pillow, wx-ocr (optional)
• Output Format — JSON structure, exit codes, boundary markers
• Coordinates — Coordinate system details
• Security Guidelines — Detailed safety practices