Firecrawl Local Skill#

Self-hosted Firecrawl integration using the v1 REST API. Tests connectivity first, executes scrape/crawl/map, handles async crawl polling automatically.

Setup (one-time)#

bash

mkdir -p ~/.openclaw/skills/firecrawl-local
cp run.sh ~/.openclaw/skills/firecrawl-local/run.sh
chmod +x ~/.openclaw/skills/firecrawl-local/run.sh

The script lives at scripts/run.sh in this skill folder — copy it into place as above.

Prerequisites: curl, jq installed. Firecrawl running at localhost:3002.

Optional env vars:

bash

export FIRECRAWL_LOCAL_URL="http://localhost:3002"  # default
export FIRECRAWL_API_KEY="fc-your-key"              # only needed if auth enabled

---

Commands#

Default — scrape a single page (URL only, no subcommand needed)#

bash

firecrawl-local https://docs.example.com/api

Scrape — explicit, with format options#

bash

firecrawl-local scrape https://docs.example.com/api
firecrawl-local scrape https://docs.example.com/api --formats markdown,html

Map — discover all URLs on a site#

bash

firecrawl-local map https://docs.example.com
firecrawl-local map https://docs.example.com --limit 200

Crawl — bulk extract multiple pages (async, auto-polled)#

bash

firecrawl-local crawl https://docs.example.com
firecrawl-local crawl https://docs.example.com --limit 30 --max-depth 2
firecrawl-local crawl https://docs.example.com --include /docs --exclude /blog

---

Agent Instructions#

When to use each command#

GoalCommandGet content from one URL (quickest)firecrawl-local <url>Discover what pages existmapGet content from one URL with format controlscrapeIngest an entire docs sitecrawlRAG pipeline ingestionmap → targeted scrape or crawl

Optimal workflows#

Documentation RAG pipeline:

text

1. map https://docs.example.com          → get full URL list
2. scrape <specific key pages>           → targeted extraction
3. Pass markdown to embedding pipeline

Full site ingestion:

text

1. crawl https://docs.example.com --limit 50 --max-depth 3
2. Results auto-polled and returned as JSON array of {url, markdown}

Parameters#

FlagApplies toDescription--limit Nmap, crawlMax pages (default: 50 for crawl, 500 for map)--max-depth NcrawlHow deep to follow links (default: 2)--include /pathcrawlOnly crawl URLs matching this path prefix--exclude /pathcrawlSkip URLs matching this path prefix--formats listscrapeComma-separated: markdown, html, rawHtml, links

Reading the output#

• scrape: Returns {success, data: {markdown, html, metadata}}
• map: Returns {success, links: [...]}
• crawl: Returns {success, data: [{url, markdown, metadata}, ...]} ← after polling completes

Failure signals and fixes#

ErrorCauseFixLocal Firecrawl unavailableService not runningStart Firecrawl, check port 3002success: falseBad URL or blockedCheck URL is reachable, try --formats htmlEmpty markdown fieldJS-rendered pageFirecrawl handles most JS — check if site blocks botsCrawl times outSite is largeReduce --limit or --max-depth

---

Script reference#

See scripts/run.sh for the full implementation. Key design decisions:

• Health check uses /health endpoint with 3s timeout
• Auth header only sent when FIRECRAWL_API_KEY is set
• Crawl polling retries every 5s up to 60 attempts (5 minutes)
• All parameters are passed via jq to prevent shell injection in JSON