OpenCrawl Skill#

Use this skill to crawl any JavaScript-rendered webpage using real Chrome browsers from a distributed worker pool. Unlike headless browser solutions (Puppeteer/Playwright), OpenCrawl requires zero local browser installation — ideal for VPS and cloud environments.

Quick Start (use our public server)#

1. Visit http://39.105.206.76:9877 and click "Register" to get a free API Key (100 credits included)

2. Set environment variables: text

   OPENCRAWL_API_KEY=ak_your_key_here
   OPENCRAWL_API_URL=http://39.105.206.76:9877
   
   

3. Start crawling!

Self-hosted (deploy your own server)#

If you prefer to run your own OpenCrawl server, see the full deployment guide: https://github.com/hlyylly/OpenCrawl

Then set OPENCRAWL_API_URL to your own server address.

---

How it works: Your request → OpenCrawl server → dispatched to a real Chrome browser worker → page rendered with full JavaScript → content extracted → uploaded to Cloudflare R2 → download URL returned to you.

Errors: On failure the script writes a JSON error to stderr and exits with code 1.

---

Tools#

1. Crawl Page#

Use this to get the full rendered text content of any webpage, including JavaScript-rendered content that simple HTTP requests cannot retrieve.

Command:

bash

python3 {baseDir}/tools/crawl.py --url "https://example.com"

Examples:

bash

# Crawl a full page
python3 {baseDir}/tools/crawl.py --url "https://www.smzdm.com/p/170177008/"

# Crawl with CSS selector to extract specific content
python3 {baseDir}/tools/crawl.py --url "https://example.com" --selector ".article-content"

# Output raw JSON response (includes downloadUrl)
python3 {baseDir}/tools/crawl.py --url "https://example.com" --raw

Optional flags:

• --selector ".css-selector" — Extract only matching elements
• --mode lite — Lite mode: no images/CSS, faster, 0.1 credit (default: full)
• --raw — Output full JSON response instead of just the text content
• --timeout 60 — Custom timeout in seconds (default: 60)

---

2. Search (Brave Search API Compatible)#

Use this to search the web using multiple search engines (DuckDuckGo + Google + Bing + Baidu) through real Chrome browsers. Returns structured results compatible with Brave Search API format.

Command:

bash

python3 {baseDir}/tools/crawl.py --search "your search query"

Examples:

bash

# Lite search — DuckDuckGo only (0.1 credit)
python3 {baseDir}/tools/crawl.py --search "python web scraping"

# Full search — 4 engines parallel (3 credits, 20-30 deduplicated results)
python3 {baseDir}/tools/crawl.py --search "python web scraping" --mode full

---

4. Check Balance#

Use this to check how many credits remain on the API key.

Command:

bash

python3 {baseDir}/tools/crawl.py --balance

---

5. Check Status#

Use this to check the OpenCrawl platform status — how many workers are online, tasks completed, etc.

Command:

bash

python3 {baseDir}/tools/crawl.py --status

---

Summary#

ActionArgumentExampleCrawl (full)--url``python3 {baseDir}/tools/crawl.py --url "https://example.com"Crawl (lite)--url --mode lite``python3 {baseDir}/tools/crawl.py --url "https://example.com" --mode liteSearch (lite)--search``python3 {baseDir}/tools/crawl.py --search "python tutorial"Search (full)--search --mode full``python3 {baseDir}/tools/crawl.py --search "python tutorial" --mode fullCheck balance--balance``python3 {baseDir}/tools/crawl.py --balanceCheck status--status``python3 {baseDir}/tools/crawl.py --status

Output: Crawl → rendered page text (or JSON with --raw). Search → JSON with web.results[] (Brave compatible). Balance → JSON. Status → JSON.

Requirements: Python 3.8+, requests library. No browser installation needed.