ClawdCursor

If a human can do it on a screen, you can too. No API? No integration? No problem.

USE AS A FALLBACK — NOT FIRST CHOICE

Before reaching for any ClawdCursor tool, ask:

1. Is there a native API? (Gmail API, GitHub API, Slack API) → use the API
2. Is there a CLI? ( git, npm, curl) → use the CLI
3. Can you edit the file directly? → do that
4. Is there a browser automation layer? (Playwright, Puppeteer) → use that

None of the above work? Now use ClawdCursor. It's for the last mile.

---

Modes at a Glance#

ModeCommandBrainTools availableserve``clawdcursor serveYou (REST client)All 42 tools via HTTPmcp``clawdcursor mcpYou (MCP client)All 42 tools via MCP stdiostart``clawdcursor startBuilt-in LLM pipelineAll 42 tools + autonomous agent

In serve and mcp modes: you reason, ClawdCursor acts. There is no built-in LLM. You call tools, interpret results, decide next steps.

---

Connecting#

Option A — REST ( clawdcursor serve)#

bash

clawdcursor serve        # starts on http://127.0.0.1:3847

All POST endpoints require: Authorization: Bearer <token> (token saved to ~/.clawdcursor/token)

text

GET  /tools              → all tool schemas (OpenAI function-calling format)
POST /execute/{name}     → run a tool: {"param": "value"}
GET  /health             → {"status":"ok","version":"0.7.5"}
GET  /docs               → full documentation

Example:

text

POST /execute/get_windows     {}
POST /execute/mouse_click     {"x": 640, "y": 400}
POST /execute/type_text       {"text": "hello world"}

If the server isn't running, start it yourself — don't ask the user:

bash

clawdcursor serve
# wait 2 seconds, then verify: GET /health

Option B — MCP ( clawdcursor mcp)#

json

{
  "mcpServers": {
    "clawdcursor": {
      "command": "clawdcursor",
      "args": ["mcp"]
    }
  }
}

Works with Claude Code, Cursor, Windsurf, Zed, or any MCP-compatible client. All 42 tools are exposed identically.

Option C — Autonomous agent ( clawdcursor start)#

text

POST /task    {"task": "Open Notepad and write Hello"}   → submit task
GET  /status  → {"status": "acting"} | "idle" | "waiting_confirm"
POST /confirm {"approved": true}                         → approve safety-gated action
POST /abort                                              → stop current task

Use delegate_to_agent tool to submit tasks from within MCP/REST sessions. Requires clawdcursor start running on port 3847.

Polling pattern:

text

POST /task  {"task": "...", "returnPartial": true}
→ poll GET /status every 2s:
    "acting"           → still running, keep polling
    "waiting_confirm"  → STOP. Ask user → POST /confirm {"approved": true}
    "idle"             → done, check GET /task-logs for result
→ if 60s+ with no progress: POST /abort, retry with simpler phrasing

returnPartial mode — send {"returnPartial": true} with POST /task: ClawdCursor skips Stage 3 (expensive vision) and returns control to you if Stage 2 fails:

json

{"partial": true, "stepsCompleted": [...], "context": "got stuck on dialog"}

You finish the task with MCP tools, then call POST /learn to save what worked.

POST /learn — adaptive learning: After completing a task with your own tool calls, teach ClawdCursor for next time:

json

POST /learn
{
  "processName": "EXCEL",
  "task": "create table with headers",
  "actions": [
    {"action": "key", "description": "Ctrl+Home to go to A1"},
    {"action": "type", "description": "Type header name"},
    {"action": "key", "description": "Tab to next column"}
  ],
  "shortcuts": {"next_cell": "Tab", "next_row": "Enter"},
  "tips": ["Use Tab between columns, Enter between rows"]
}

This enriches the app's guide JSON. Stage 2 reads it on the next run — no vision fallback needed.

---

The Universal Loop#

Every GUI task follows the same pattern regardless of transport:

text

1. ORIENT  →  read_screen() or get_windows()          see what's open and focused
2. ACT     →  smart_click() / smart_type() / key_press()   do the thing
3. VERIFY  →  check return value → window state → text check → screenshot
4. REPEAT  →  until done

Verification (cheapest to most expensive)#

1. Tool return value — every tool reports success/failure. Check it first.
2. Window state — get_active_window(), get_windows() — did a dialog appear? Did the title change?
3. Text check — read_screen() or smart_read() — is the expected text visible?
4. Screenshot — desktop_screenshot() — only when text methods fail. Costs the most.
5. Negative check — look for error dialogs, wrong window, unchanged screen.

Always verify after: sends, saves, deletes, form submissions. Skip verification for: mid-sequence keystrokes, scrolling.

---

Tool Decision Trees#

Perception — always start here#

text

read_screen()          → FIRST. Accessibility tree: buttons, inputs, text, with coords.
                          Fast, structured, works on native apps.
ocr_read_screen()      → When a11y tree is empty (canvas UIs, image-based apps).
smart_read()           → Combines OCR + a11y. Good first call when unsure.
desktop_screenshot()   → LAST RESORT. Only when you need pixel-level visual detail.
desktop_screenshot_region(x,y,w,h) → Zoomed crop when you need detail in one area.

Clicking#

text

smart_click("Save")              → FIRST. Finds by label/text via OCR + a11y, clicks.
                                   Pass processId to target the right window.
invoke_element(name="Save")      → When you know the exact automation ID from read_screen.
cdp_click(text="Submit")         → Browser elements. Requires cdp_connect() first.
mouse_click(x, y)                → LAST RESORT. Raw coordinates from a screenshot.

Typing#

text

smart_type("Email", "user@x.com")  → FIRST. Finds field by label, focuses, types.
cdp_type(label="Email", text="…")  → Browser inputs. Requires cdp_connect() first.
type_text("hello")                 → Clipboard paste into whatever is focused.
                                     Use after manually focusing with smart_click.

Browser / CDP#

text

1. navigate_browser(url)     → opens URL, auto-enables CDP
2. cdp_connect()             → connect to browser DevTools Protocol
3. cdp_page_context()        → list interactive elements on page
4. cdp_read_text()           → extract DOM text (returns empty on canvas apps → use OCR)
5. cdp_click(text="…")       → click by visible text
6. cdp_type(label, text)     → fill input by label
7. cdp_evaluate(script)      → run JavaScript in page context
8. cdp_scroll(direction, px) → scroll page via DOM (not mouse wheel)
9. cdp_list_tabs()           → list all open tabs
10. cdp_switch_tab(target)   → switch to a specific tab

If CDP isn't connected, switch tabs with keyboard:

text

key_press("ctrl+1")          → tab 1
key_press("ctrl+tab")        → next tab
key_press("ctrl+shift+tab")  → previous tab

Window Management#

text

get_windows()                         → list all open windows (use to find PIDs)
get_active_window()                   → what's in the foreground right now
focus_window(processName="Discord")   → bring to front (auto-minimizes phantom off-screen windows)
minimize_window(processName="calc")   → minimize a window — 1 call, cross-platform
                                        also accepts: processId, title

Rule: Always focus_window() before key_press() or type_text(). Keystrokes go to whatever has focus — if that's your terminal, not the target app.

Canvas apps (Google Docs, Figma, Notion)#

DOM has no readable text. Pattern:

text

ocr_read_screen()          → read content (DOM extraction fails)
mouse_click(x, y)          → click into the canvas area
type_text("your text")     → clipboard paste works even on canvas

---

Quick Patterns#

Open app and type:

text

open_app("notepad") → wait(2) → smart_read() → type_text("Hello") → smart_read()

Read a webpage:

text

navigate_browser(url) → wait(3) → cdp_connect() → cdp_read_text()

Fill a web form:

text

cdp_connect() → cdp_type("Email", "x@x.com") → cdp_type("Password", "…") → cdp_click("Submit")

Cross-app copy/paste:

text

focus_window("Chrome") → key_press("ctrl+a") → key_press("ctrl+c")
→ read_clipboard() → focus_window("Notepad") → type_text(clipboard)

Send email via Outlook:

text

open_app("outlook") → wait(2) → smart_click("New Email")
→ mouse_click(to_field_x, to_field_y) → type_text("recipient@x.com") → key_press("Tab")
→ mouse_click(subject_x, subject_y) → type_text("Subject") → key_press("Tab")
→ mouse_click(body_x, body_y) → type_text("Body text")
→ mouse_click(send_x, send_y)

Autonomous complex task (requires clawdcursor start):

text

delegate_to_agent("Open Gmail, find latest email from Stripe, forward to billing@x.com")
→ poll GET /status every 2s
→ if waiting_confirm: ask user → POST /confirm {"approved": true}
→ if idle: task done

---

Full Tool Reference (42 tools)#

Speed: ⚡ Free/instant · 🔵 Cheap · 🟡 Moderate · 🔴 Vision (expensive)

Perception (6)#

ToolWhat it doesWhenread_screenA11y tree — buttons, inputs, text, coords⚡ Default first readsmart_readOCR + a11y combined🔵 When unsure which to useocr_read_screenRaw OCR text with bounding boxes🔵 Canvas UIs, empty a11y treesdesktop_screenshotFull screen image (1280px wide)⚡ Last resort visual checkdesktop_screenshot_regionZoomed crop of specific area⚡ Fine-grained visual detailget_screen_sizeScreen dimensions and DPI⚡ Coordinate calculations

Mouse (7)#

ToolWhat it doesWhensmart_clickFind element by text/label, click🔵 First choice for clickingmouse_clickLeft click at (x, y)⚡ Last resortmouse_double_clickDouble click at (x, y)⚡ Open files, select wordsmouse_right_clickRight click at (x, y)⚡ Context menusmouse_hoverMove cursor without clicking⚡ Hover menusmouse_scrollScroll at position (physical mouse wheel)⚡ Scroll contentmouse_dragDrag from start to end — accepts startX/startY/endX/endY or x1/y1/x2/y2⚡ Resize, select ranges

Keyboard (5)#

ToolWhat it doesWhensmart_typeFind input by label, focus it, type🔵 First choice for form fieldstype_textClipboard paste into focused element⚡ After manually focusingkey_pressSend key combo ( ctrl+s, Return, alt+tab)⚡ After focus_windowshortcuts_listList keyboard shortcuts for current app⚡ Before reaching for mouseshortcuts_executeRun a named shortcut (fuzzy match)⚡ Save, copy, paste, undo

Window Management (5)#

ToolWhat it doesWhenget_windowsList all open windows with PIDs and bounds⚡ Situational awarenessget_active_windowCurrent foreground window⚡ Check current focusget_focused_elementElement with keyboard focus⚡ Debug wrong-field typingfocus_windowBring window to front (auto-clears off-screen phantoms)⚡ Always before key_pressminimize_windowMinimize by processName, processId, or title⚡ Clear focus stealers

UI Elements (2)#

ToolWhat it doesWhenfind_elementSearch UI tree by name or type⚡ Find automation IDsinvoke_elementInvoke element by automation ID or name⚡ When ID known from read_screen

Clipboard (2)#

ToolWhat it doesWhenread_clipboardRead clipboard text⚡ After copy operationswrite_clipboardWrite text to clipboard⚡ Before paste operations

Browser / CDP (11)#

ToolWhat it doesWhencdp_connectConnect to browser DevTools Protocol⚡ First step for any browser taskcdp_page_contextList interactive elements on page⚡ After connectcdp_read_textExtract DOM text⚡ Read page contentcdp_clickClick by CSS selector or visible text⚡ Browser clickscdp_typeType into input by label or selector⚡ Browser form fillingcdp_select_optionSelect dropdown option⚡ Select elementscdp_evaluateRun JavaScript in page context⚡ Custom queriescdp_scrollScroll page via DOM ( direction, amount px)⚡ DOM-level scrollcdp_wait_for_selectorWait for element to appear⚡ After navigation/AJAXcdp_list_tabsList all browser tabs⚡ When on wrong tabcdp_switch_tabSwitch to a tab by title or index⚡ After cdp_list_tabs

Orchestration (4)#

ToolWhat it doesWhenopen_appLaunch application by name⚡ First step for desktop tasksnavigate_browserOpen URL (auto-enables CDP)⚡ First step for browser taskswaitPause N seconds⚡ After opening apps, let UI renderdelegate_to_agentSend task to built-in autonomous agent🟡 Complex multi-step tasks (requires clawdcursor start)

---

Provider Setup (agent mode only)#

ProviderSetupCostOllama (local)ollama pull qwen2.5:7b && ollama serve$0 — fully offline, no data leaves machineAny cloudSet env var: ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, MOONSHOT_API_KEY, etc.VariesOpenClaw usersAuto-detected from ~/.openclaw/agents/main/auth-profiles.jsonNo extra setup

Run clawdcursor doctor to auto-detect and validate providers.

---

Security#

• Network isolation: Binds to 127.0.0.1 only. Verify: netstat -an | findstr 3847 — should show 127.0.0.1:3847, never 0.0.0.0:3847
• Ollama: 100% offline. Screenshots stay in RAM, never leave the machine.
• Cloud providers: Screenshots/text sent only to your configured provider. No telemetry, no analytics, no third-party logging.
• Token auth: All mutating POST endpoints require Authorization: Bearer <token>. Token at ~/.clawdcursor/token.
• Safety tiers: Auto / Preview / Confirm. Agents must never self-approve Confirm actions.

---

Coordinate System#

All mouse tools use image-space coordinates from a 1280px-wide viewport — matching screenshots from desktop_screenshot. DPI scaling is handled automatically. Do not pre-scale coordinates.

---

Safety#

TierActionsBehavior🟢 AutoNavigation, reading, opening appsRuns immediately🟡 PreviewTyping, form fillingLogged🔴 ConfirmSend, delete, purchasePauses — always ask user first

• Never self-approve Confirm actions.
• Alt+F4 and Ctrl+Alt+Delete are blocked.
• Server binds to 127.0.0.1 only.
• First run requires explicit user consent for desktop control.

---

Error Recovery#

ProblemFixPort 3847 not respondingclawdcursor serve — wait 2s — GET /health401 UnauthorizedToken changed — read ~/.clawdcursor/token and use fresh valueCDP not availableChrome must be open. navigate_browser(url) auto-enables it.CDP on wrong tabcdp_list_tabs() → cdp_switch_tab(target)``focus_window failsget_windows() to confirm title/processName, then retrysmart_click can't find elementread_screen() for coords → mouse_click(x, y)``key_press goes to wrong windowYou skipped focus_window — always focus firstcdp_read_text returns emptyCanvas app — use ocr_read_screen() insteadSame action fails 3+ timesTry a completely different approach

---

Platform Support#

PlatformA11yOCRCDPWindows (x64/ARM64)PowerShell + .NET UIAWindows.Media.OcrChrome/EdgemacOS (Intel/Apple Silicon)JXA + System EventsApple VisionChrome/EdgeLinux (x64/ARM64)AT-SPITesseractChrome/Edge

macOS: Grant Accessibility in System Settings → Privacy → Accessibility. Linux: sudo apt install tesseract-ocr for OCR support.