Web + Desktop Automation#

Use this skill when a task may involve:

• Opening or controlling websites
• Reading or extracting page content
• Filling forms, clicking buttons, logging in
• Downloading or uploading files
• Controlling desktop apps with mouse/keyboard
• Combining browser steps with local app steps

Core rule#

Prefer the simplest reliable path:

1. If the task can be done in the browser, use browser automation.
2. If the task needs local apps or OS-level interaction, use desktop automation.
3. If both are needed, split the job into clear phases and verify after each phase.

Execution strategy#

1) Classify the task#

Decide which of these applies:

• Browser only
• Desktop only
• Mixed browser + desktop

2) Browser automation#

Use browser automation for:

• Navigation
• Search
• Page reading
• Form filling
• Clicking controls
• File upload/download
• Logged-in web workflows

Prefer stable selectors and explicit waits. Avoid brittle coordinate-based clicking when browser selectors exist.

3) Desktop automation#

Use desktop automation for:

• Native apps
• Window switching
• Copy/paste between apps
• File manager operations
• UI flows outside the browser

Prefer application/window-aware methods when available. Use image-based or coordinate-based control only when necessary.

4) Mixed workflows#

Break the task into phases:

• Browser phase
• Desktop phase
• Browser phase again if needed

After each phase, verify the result before continuing.

Recovery rules#

If a step fails:

1. Re-check the current UI state
2. Re-locate the target element or window
3. Try a more stable selector or a different interaction method
4. If the task risks loss of data or irreversible action, stop and ask the user

Best practices#

• Prefer deterministic steps over guessing
• Avoid rapid blind retries
• Capture key state when tasks are long or fragile
• Keep flows small and modular
• Use scripts for repeated actions
• Use scripts/browser_runner.py for Playwright browser automation templates
• Use scripts/desktop_runner.py for PyAutoGUI desktop automation templates
• Use scripts/mixed_orchestrator.py for browser + desktop handoffs
• Put browser-specific patterns in references/browser-workflows.md
• Put desktop-specific patterns in references/desktop-workflows.md
• Put mixed-flow orchestration examples in references/mixed-flows.md
• Put dependency and installation notes in references/dependencies.md
• Put a realistic browser-download → desktop-edit → browser-upload flow in references/mixed-example.md
• See requirements.txt for a minimal install set
• Put dependency and installation notes in references/dependencies.md
• Put a realistic browser-download → desktop-edit → browser-upload flow in references/mixed-example.md
• Put dependency and installation notes in references/dependencies.md