Firecrawl Skill#

Powerful web scraping powered by Firecrawl - turn websites into LLM-ready markdown.

Overview#

Firecrawl provides APIs for:

• Scrape - Single page extraction to markdown
• Crawl - Entire site crawling with depth control
• Map - URL discovery from a starting point
• Batch - Multiple URL processing
• Extract - Structured data extraction with schemas

Prerequisites#

1. Firecrawl API Key - Get free tier at https://firecrawl.dev
2. Install Python dependencies: requests

Configuration#

Set environment variable:

bash

export FIRECRAWL_API_KEY="fc-your-api-key"

Usage#

Single Page Scraping#

bash

# Basic scrape
firecrawl scrape https://example.com

# With specific options
firecrawl scrape https://example.com --formats markdown,html --only-main-content

# Wait for JS rendering
firecrawl scrape https://spa-app.com --wait-for 2000

Site Crawling#

bash

# Crawl entire site (up to limit)
firecrawl crawl https://docs.example.com --limit 50

# With depth control
firecrawl crawl https://blog.example.com --max-depth 2 --limit 100

# Include/exclude patterns
firecrawl crawl https://site.com --include "/blog/*" --exclude "/admin/*"

# Custom formats
firecrawl crawl https://docs.example.com --formats markdown,links

URL Mapping#

bash

# Discover all URLs from a site
firecrawl map https://example.com

# With search term
firecrawl map https://docs.python.org --search "tutorial"

Batch Processing#

bash

# Scrape multiple URLs
firecrawl batch urls.txt --output ./scraped/

# From JSON list
firecrawl batch urls.json --formats markdown --concurrency 5

Structured Extraction#

bash

# Extract specific data using CSS selectors
firecrawl extract https://example.com/products \
  --schema '{"name": ".product-title", "price": ".price", "description": ".desc"}'

# Extract to JSON
firecrawl extract https://news.example.com/article --schema article-schema.json

Output Formats#

Markdown#

Clean, LLM-ready markdown with:

• Headings preserved
• Links converted to markdown format
• Images with alt text
• Tables formatted as markdown tables

HTML#

Raw or cleaned HTML

Links#

Extracted link lists for further crawling

Screenshot#

Page screenshot (if requested)

Use Cases#

Knowledge Base Building#

bash

# Crawl documentation site
firecrawl crawl https://docs.framework.com --limit 200 -o ./kb/

# Merge into single file for RAG
cat ./kb/*.md > knowledge-base.md

Research & Analysis#

bash

# Scrape competitor pricing
firecrawl batch competitors.txt --extract pricing-schema.json

# Monitor blog updates
firecrawl map https://blog.company.com --since 2024-01-01

Content Migration#

bash

# Export old CMS content
firecrawl crawl https://old-site.com --formats markdown,html -o ./export/

Scripts#

All functionality via scripts/firecrawl.py:

• Handles API authentication
• Automatic rate limiting
• Retry logic for failures
• Progress tracking for large crawls

Integration#

Works well with:

• markdown-sync-pro - Sync scraped content to Notion/GitHub
• arxiv-paper - Combine with academic paper downloads
• maybe-finance - Scrape financial data for analysis