Screen Vision#

Control the desktop visually: screenshot → AI vision analysis → execute actions → loop until done.

Quick Start#

1. Setup (one-time)#

Detect platform and install dependencies:

bash

bash scripts/setup/setup-linux.sh --headless   # Linux server (no desktop)
bash scripts/setup/setup-linux.sh --desktop     # Linux with desktop
bash scripts/setup/setup-mac.sh                 # macOS
python scripts/setup/setup-win.py          # Windows

2. Configure API#

Copy config.example.json to config.json and fill in your vision API credentials. You must set baseUrl, apiKey, and model — supports any OpenAI-compatible API.

json

{
  "vision": {
    "baseUrl": "https://api.siliconflow.cn/v1",
    "apiKey": "sk-your-key",
    "model": "Qwen/Qwen3-VL-32B"
  }
}

Environment variables also work: SV_VISION_API_KEY, SV_VISION_BASE_URL, SV_VISION_MODEL. See references/API_CONFIG.md for all supported providers and detailed setup.

3. Usage#

The skill operates through a screenshot-analyze-action loop:

1. Take screenshot → bash scripts/platform/screenshot.sh [output_path] [display]
2. Analyze with AI → python3 scripts/vision/analyze.py --image <path> --task "<task>"
3. Execute action → python3 scripts/platform/execute.py --action <type> [options]
4. Full task loop → python3 scripts/core/run_task.py --task "<task>"

Architecture#

text

User task → run_task.py (orchestrator)
  ├── screenshot.sh (capture screen)
  ├── diff_check.py (detect changes, skip if unchanged → saves tokens)
  ├── analyze.py (send screenshot + task to vision API)
  ├── safety_check.py (block dangerous operations)
  ├── execute.py (xdotool/cliclick/pyautogui)
  └── loop until done or timeout

Platform Tools#

PlatformScreenshotMouse/KeyboardNotesLinuxscrotxdotoolHeadless: XFCE4 + VNCmacOSscreencapturecliclickNeeds Accessibility permissionWindowspyautoguipyautoguiNo extra setup needed

See references/PLATFORM_GUIDE.md for platform-specific commands.

Vision Providers#

Supports any OpenAI-compatible vision API. You choose the provider and model.

Recommended Models#

ModelProviderCost/TaskQualityQwen3-VL-32BSiliconFlowLow★★★★GLM-4V-PlusZhipu BigModelLow★★★★GPT-5.4-MiniOpenAI / relaysMedium★★★★★GPT-5.4 CUAOpenAIHigh★★★★★Llama 3.2 VisionOllama (local)Free★★

See references/API_CONFIG.md for per-provider configuration examples.

No defaults are hardcoded — you must configure your own API credentials before use.

Action Types#

• click — Click at (x, y). Supports left/right/double-click.
• type — Type text string.
• key — Press a key (Return, Tab, Escape, etc.).
• scroll — Scroll up or down.
• drag — Drag from (x1,y1) to (x2,y2).
• wait — Wait for screen to update.
• done — Task complete.
• failed — Cannot complete task.

Safety#

• Blocked: rm -rf, format disk, shutdown, drop database, etc.
• Confirmation required: delete, sudo, payment-related operations
• Limits: max 5 minutes, max 100 actions per task
• Logging: all screenshots saved to /tmp/screen-vision/logs/
• Auto-stop on error or API failure

Examples#

See references/EXAMPLES.md for usage examples.

Config#

VariableDefaultDescriptionSV_VISION_API_KEY—Vision API keySV_VISION_BASE_URL—API endpoint (required)SV_VISION_MODEL—Vision model name (required)SV_DISPLAY``:1X11 display (Linux)SV_MAX_DURATION``5Max task duration (min)SV_MAX_ACTIONS``100Max actions per taskSV_SCREENSHOT_INTERVAL``1.0Seconds between screenshots