Auto Scraping to CSV — Agent-Driven Web Scraping#

Scrape any webpage using text-based DOM manipulation and export structured data to CSV. The agent handles complex page nuances — infinite scroll, pagination, popups, lazy loading — and asks clarifying questions when data is ambiguous. No external LLM required.

Philosophy: Let the Agent Figure It Out#

Traditional scraping requires you to inspect HTML, write CSS selectors, handle edge cases, and debug when the site changes. This skill flips that:

You say what you want. The agent handles the how.

text

You: "Scrape product catalog"
Agent: "I see 50 products across 5 pages with infinite scroll.
        I also see 'Price', 'Sale Price', and 'Member Price' columns.
        Which price field should I extract?"
You: "Sale Price"
Agent: [handles scrolling, pagination, extracts 50 rows] → products.csv

The agent will:

1. Explore the page structure via text-based DOM
2. Detect complexity — scroll, pagination, tabs, filters
3. Ask questions when ambiguous (multiple price fields, missing data, format choices)
4. Handle edge cases — dismiss popups, wait for lazy loading, retry on errors
5. Export CSV — clean, structured, ready to use

When to Use#

• Product catalogs: "Scrape all laptops with prices and ratings"
• News/articles: "Get latest blog posts with titles, dates, authors"
• Directory listings: "Extract all company names, emails, and websites"
• Table data: "Get the pricing table from this SaaS page"
• Real estate: "Scrape listings with price, beds, baths, square footage"
• Job boards: "Get job titles, companies, locations, and salary ranges"
• Social feeds: "Extract posts with engagement counts (likes, comments, shares)"
• Research data: "Get citation counts, authors, publication dates from this index"

How It Works#

text

Claude (Host Model)
    ↕  HTTP
Bridge Server (Node.js + Playwright)
    ↕  page.evaluate()
Browser (Chromium) ← Page-Agent injected

1. Bridge launches a local Chromium browser via Playwright

2. Page-Agent is injected as an IIFE script from CDN into the target page

3. Page-Agent indexes the DOM and generates a simplified text representation: text

   [5]<a >Widget Pro Laptop /></a>
   [12]<div >$1,299.99 /></div>
   [18]<div >4.5 stars (128 reviews) /></div>
   
   

4. Claude receives the text state, understands the page structure, decides actions

5. Agent asks questions when data is ambiguous or complex

6. Loop continues until all data is extracted or user says stop

7. CSV export converts structured JSON to CSV

---

First-Time Setup#

1. Install Playwright#

bash

npm install -D playwright
npx playwright install chromium

2. Place the Bridge Script#

bash

# Copy from this skill folder to your agents directory
cp .claude/skills/auto-scraping-to-csv/page-agent-bridge.mjs .claude/agents/

3. Start the Bridge#

bash

node .claude/agents/page-agent-bridge.mjs

Default port: 9876.

4. Verify Health#

bash

curl http://localhost:9876/health

---

Agent-Driven Scraping Workflow#

Step 1: Request What You Want#

text

/scrape-to-csv https://example.com/products
  "Extract all products with name, price, rating, and number of reviews"

Step 2: Agent Explores and Asks Questions#

The agent will fetch the DOM state and may ask:

text

Agent: "I found 24 products on the current page. The site has infinite scroll
        and I can see more products loading as I scroll.
        How many products do you want total? (type 'all' for everything)"
You: "50"

Agent: "I see three price fields:
        - Original Price: $1,299
        - Sale Price: $999
        - Member Price: $899
        Which one should I extract?"
You: "Sale Price"

Agent: "Some products are out of stock and show 'N/A' for price.
        Should I:
        A) Include them with 'N/A'
        B) Skip those rows
        C) Mark as 'Out of Stock' in a separate column"
You: "C"

Step 3: Agent Handles Complexity#

The agent automatically:

• Scrolls to load more content
• Clicks pagination buttons
• Waits for lazy-loaded images/data
• Dismisses cookie banners and popups
• Retries if elements are slow to appear
• Stops gracefully if the site blocks scraping

Step 4: Preview and Confirm#

text

Agent: "Extracted 50 products. Here's a preview:

| name | price | rating | reviews | stock_status |
|------|-------|--------|---------|-------------|
| Widget Pro | $999 | 4.5 | 128 | In Stock |
| Gadget Mini | $499 | 4.2 | 89 | In Stock |
| ... | ... | ... | ... | ... |

Save as 'products.csv'? (yes/no/change columns)"
You: "yes"

Step 5: CSV Delivered#

text

Agent: "Saved to ./products.csv — 50 rows, 5 columns"

---

Handling Complex Scenarios#

Infinite Scroll#

text

You: "Scrape all posts from this feed"
Agent: "The page loads 10 posts at a time via infinite scroll.
        I've scrolled 3 times and found 30 posts so far.
        Should I continue scrolling? (yes/no/scroll N more times)"
You: "scroll 5 more times"
Agent: [scrolls] → "Found 80 total posts. Saved to feed.csv"

Pagination#

text

You: "Get all products from this catalog"
Agent: "I see pagination: Page 1 of 12.
        I can scrape all 12 pages (may take 2-3 minutes) or a subset.
        How many pages?"
You: "all"
Agent: [clicks through 12 pages] → "360 products saved to catalog.csv"

Login Required#

text

You: "Scrape my order history"
Agent: "The page redirects to a login screen.
        I need credentials to proceed.
        Please provide:
        - Username/email
        - Password
        Or type 'skip' to abort."
You: "skip"
Agent: "Aborted. No data extracted."

Multiple Data Formats#

text

You: "Get prices from this page"
Agent: "I found prices in multiple formats:
        - '$19.99' (US format)
        - '€17.99' (EU format)
        - '¥2,980' (JP format)
        Which format should I standardize to?"
You: "Keep as-is, add a 'currency' column"
Agent: [extracts with currency detection] → prices.csv

Missing / Inconsistent Data#

text

You: "Extract employee data from this directory"
Agent: "I see 100 employee cards. Some are missing:
        - 12 missing phone numbers
        - 8 missing job titles
        - 3 missing photos
        How should I handle missing fields?
        A) Leave blank
        B) Fill with 'N/A'
        C) Skip those rows entirely"
You: "B"
Agent: [extracts 100 rows with 'N/A' for missing fields] → employees.csv

---

Natural Language Commands#

/scrape-to-csv <url> <description>#

General scraping with CSV export.

text

/scrape-to-csv https://news.ycombinator.com
  "Get top 30 stories with title, URL, points, and comment counts"

/scrape-to-csv https://www.anthropic.com/news
  "Latest blog posts: title, date, category, URL"

/scrape-to-csv https://example.com/realestate
  "Listings: address, price, beds, baths, sqft, listing agent"

/scrape-table <url> <selector_or_description>#

Extract a specific table.

text

/scrape-table https://example.com/pricing
  "The comparison table with Basic/Pro/Enterprise columns"

/scrape-table https://example.com/sales
  "Q4 2024 revenue breakdown table"

/scrape-news <url>#

Optimized for news/blog scraping.

text

/scrape-news https://techcrunch.com
  "Latest 20 articles: title, author, date, excerpt, URL"

/scrape-news https://blog.openai.com
  "All posts from 2024: title, date, tags, URL"

/scrape-products <url>#

Optimized for e-commerce.

text

/scrape-products https://amazon.com/s?k=laptops
  "Laptops: name, brand, price, rating, prime eligible, URL"

/scrape-products https://shopify-store.com/collections/all
  "All products: name, price, compare-at price, availability"

---

Output Format#

The agent produces a structured markdown report:

markdown

## Scraping Report — example.com/products
**Session:** a1b2c3d4 | **Duration:** 2m 14s | **Rows:** 50

### Task
Extract all products with name, price, rating, and number of reviews

### Agent Decisions
- **Pagination**: Detected infinite scroll, scrolled 5 times
- **Price field**: Chose "Sale Price" per user request
- **Missing data**: Filled out-of-stock prices with "N/A" per user request
- **Columns**: name, sale_price, rating, review_count, stock_status

### Sample Data

| name | sale_price | rating | review_count | stock_status |
|------|-----------|--------|-------------|-------------|
| Widget Pro | $999 | 4.5 | 128 | In Stock |
| Gadget Mini | $499 | 4.2 | 89 | In Stock |
| Super Gizmo | $1,299 | 4.8 | 256 | Out of Stock |

### File
`./products.csv` — 50 rows, 5 columns

---

CSV Conversion Options#

Option A — Python (recommended)#

python

import json, csv, re

# Bridge returns: "✅ Executed JavaScript. Result: [{...}, {...}]"
msg = """PASTE_BRIDGE_RESPONSE_HERE"""
match = re.search(r'Result: (\[.*\])', msg)
if match:
    data = json.loads(match.group(1))
    with open('output.csv', 'w', newline='') as f:
        writer = csv.DictWriter(f, fieldnames=data[0].keys())
        writer.writeheader()
        writer.writerows(data)
    print(f"Wrote {len(data)} rows to output.csv")

Option B — Node.js#

javascript

const fs = require('fs');
const data = JSON.parse(fs.readFileSync('data.json', 'utf8'));
const headers = Object.keys(data[0]);
const csv = [
  headers.join(','),
  ...data.map(row => headers.map(h => `"${(row[h]||'').replace(/"/g,'""')}"`).join(','))
].join('\n');
fs.writeFileSync('output.csv', csv);

---

Troubleshooting#

Bridge won't start#

text

Error: Cannot find module 'playwright'

Fix: npm install -D playwright && npx playwright install chromium

Site blocks scraping#

Agent detects: "The site returned 403 Forbidden. This may be bot protection." Options:

• Try headless: false (looks more like a real user)
• Add delays between requests
• Use a different user agent

Page loads but no data found#

Agent detects: "The page loaded but I see mostly navigation elements. Content may be behind a login or loaded dynamically." Agent asks: "Should I wait longer, scroll down, or do you have login credentials?"

Data looks wrong#

Agent detects: "Prices show as 'NaN' or empty. The site may use JavaScript to render prices." Agent asks: "Should I try executing JavaScript to extract the real values, or skip this field?"

---

Comparison with Other Tools#

ToolSetupSelectorsComplex PagesAgent QuestionsBest ForAuto Scraping to CSVnpm installNone neededHandles automaticallyYes, clarifies ambiguityOne-off extraction, exploratory scrapingBeautifulSouppip installRequiredManual handlingNoKnown structure, repeated scrapingScrapyProject setupRequiredMiddleware neededNoLarge-scale crawling, pipelinesPlaywright E2Enpm installRequiredManual handlingNoTesting, automationBrowser-UseAPI keyNonePartialLimitedMulti-page research

Use this skill when:

• You want to scrape without writing selectors
• The page structure is complex or unknown
• You need the agent to handle edge cases (scroll, popups, pagination)
• You want clarifying questions when data is ambiguous
• You need quick one-off extraction to CSV

---

Bridge API Reference#

POST /sessions#

Launch a new browser session.

Body:

json

{ "url": "https://example.com", "headless": false }

Response: { "id": "abc123", "url": "https://example.com" }

GET /sessions/:id/state#

Get text-based DOM state.

Response: { url, title, header, content, footer }

POST /sessions/:id/act#

Execute an action.

Body:

json

{ "action": "executeJavascript", "params": { "script": "return document.title;" } }

DELETE /sessions/:id#

Close session.

POST /shutdown#

Stop bridge.

---

Skill: auto-scraping-to-csv v1.0.0 | Bridge: page-agent-bridge.mjs | Powered by Alibaba Page-Agent + Playwright