PDF Miner Skill#

Extract text and tables from PDF files using pdfplumber (global market formats).

Prerequisites#

bash

python -m pip install pdfplumber

For OCR capabilities (scanned/image PDFs), also install:

bash

python -m pip install pymupdf openai

Initial Setup for OCR#

Before using --ocr, you must provide a vision API credential. There are three ways:

1. Environment variables (recommended for temporary use): bash

   export OCR_API_KEY="your-openrouter-api-key"
   export OCR_MODEL="qwen/qwen3.6-plus:free"   # optional
   export OCR_BASE_URL="https://openrouter.ai/api/v1"   # optional
   
   

2. Config file (persistent, skill-specific):

   Create skills/skills/pdf-miner/config.json with: json

   {
     "vision_api_key": "your-openrouter-api-key",
     "vision_model": "qwen/qwen3.6-plus:free",
     "vision_base_url": "https://openrouter.ai/api/v1"
   }
   
   

3. Command-line arguments (override per invocation): bash

   python scripts/extract_pdf.py scanned.pdf --ocr --ocr-api-key "sk-..." --ocr-model "stepfun/step-3.5-flash:free"
   
   

Usage#

Run commands from this skill directory.

Basic Extraction#

bash

# Full extraction (text + tables)
python scripts/extract_pdf.py input.pdf

# Output to custom path
python scripts/extract_pdf.py input.pdf output.md

# Specific pages
python scripts/extract_pdf.py input.pdf --pages 1-5,10,15-20

# Text or tables only
python scripts/extract_pdf.py input.pdf --text-only
python scripts/extract_pdf.py input.pdf --tables-only
python scripts/extract_pdf.py input.pdf --tables-only --json

Advanced Modes#

bash

# Search: find pages containing keywords with context
python scripts/extract_pdf.py report.pdf --search "Vietnam export penetration"

# Metrics: extract lines with keywords + numeric values
python scripts/extract_pdf.py report.pdf --metrics "market size growth export penetration"

# TOC: extract table of contents / chapter structure (robust, multi-format)
python scripts/extract_pdf.py report.pdf --toc
# Optionally adjust sensitivity (default: 3 entries per page required)
python scripts/extract_pdf.py report.pdf --toc --toc-min-entries 2

# Diff: compare two PDFs, show pages unique to each
python scripts/extract_pdf.py old_version.pdf new_version.pdf --diff

# Chunk: split output into LLM-friendly chunks
python scripts/extract_pdf.py report.pdf --chunk             # single file, 8000 chars each
python scripts/extract_pdf.py report.pdf --chunk --max-chars 4000
python scripts/extract_pdf.py report.pdf --chunk --output-dir ./chunks   # separate files

# Clean headers/footers
python scripts/extract_pdf.py report.pdf --clean-headers

# Batch: process multiple PDFs
python scripts/extract_pdf.py file1.pdf file2.pdf file3.pdf --output-dir ./extracted

OCR for Scanned/Image PDFs (Automatic by Default)#

OCR is automatically triggered for pages with very little extractable text (default threshold: 100 characters). This helps handle scanned or image-based PDFs without requiring the --ocr flag.

Usage Examples

bash

# Automatic OCR (default behavior)
python scripts/extract_pdf.py scanned.pdf

# Force OCR on all pages (ignore text length)
python scripts/extract_pdf.py scanned.pdf --ocr

# Force OCR only on specific pages
python scripts/extract_pdf.py scanned.pdf --ocr --ocr-pages 1-5,10

# Adjust OCR quality (DPI)
python scripts/extract_pdf.py scanned.pdf --ocr --ocr-dpi 300

# Use a different vision model
python scripts/extract_pdf.py scanned.pdf --ocr --ocr-model "stepfun/step-3.5-flash:free"

# Disable automatic OCR detection (if you want pure extraction only)
python scripts/extract_pdf.py file.pdf --no-auto-ocr

# Change the low-text threshold (default 100 chars)
python scripts/extract_pdf.py file.pdf --ocr-threshold 200

Configuration

OCR requires a vision API key. See Initial Setup for OCR.

OptionDefaultDescription--ocroffForce OCR on pages (with auto-detect or --ocr-pages)--auto-ocronAutomatically OCR low-text pages (hidden; use --no-auto-ocr to disable)--no-auto-ocr-Disable automatic OCR detection--ocr-pages-Comma-separated pages/ranges to OCR (requires --ocr)--ocr-threshold100Minimum text length to consider a page as "sufficient" (characters)--ocr-dpi200Image DPI for OCR rendering--ocr-api-keyfrom env/configOverride API key--ocr-base-urlfrom env/configOverride API base URL--ocr-modelfrom env/configOverride vision model

Troubleshooting

OCR failed with "No API key"

→ Configure your API key in config.json or via OCR_API_KEY env var.

OCR model rejects images

→ The configured model might not support vision. Choose a vision-capable model (e.g., qwen/qwen3.6-plus:free, stepfun/step-3.5-flash:free). The script will attempt to auto-fallback to a known good model if the configured one lacks vision support.

Too many pages being OCR'd

→ Increase the threshold: --ocr-threshold 300 or --no-auto-ocr and selectively use --ocr-pages.

Rate limit errors

→ Reduce concurrent OCR calls, switch to a paid model tier, or try a different provider.

Configuration Reference#

OptionDefaultSourceOCR_API_KEY(none)env OCR_API_KEY or config.json vision_api_key``OCR_MODEL``qwen/qwen3.6-plus:freeenv OCR_MODEL or config.json vision_model``OCR_BASE_URL``https://openrouter.ai/api/v1env OCR_BASE_URL or config.json vision_base_url

Precedence: CLI argument > environment variable > config.json > hardcoded default.

Tool Comparison#

ToolPDFGlobal textTablesSearchMetricsDiffChunkweb_fetch❌-❌❌❌❌❌scrapling❌-❌❌❌❌❌pypdf⚠️ garbled❌❌❌❌❌❌pdfplumber✅✅✅✅✅✅✅

Modes Reference#

ModeFlagWhat it doesFull(default)Extract all text + tables, page by pageSearch--search "kw1 kw2"Find pages with keywords, show ±N lines context (default 5)Metrics--metrics "kw1 kw2"Extract lines with keywords AND numeric dataTOC--tocDetect table of contents / chapter structure (robust multi-format)--toc-min-entries NMinimum TOC entries per page to trust detection (default: 3)Diff--diffCompare two PDFs, show matched vs unique pagesChunk--chunkSplit into LLM-friendly pieces ( --max-chars N)Clean--clean-headersAuto-detect and remove repeated header/footer linesBatchfile1 file2 ...Process multiple PDFs, output to --output-dir

Output Options#

FlagEffect--output-dir ./dirOutput to specified directory--chunk --output-dirEach chunk as separate file--context NContext lines around search matches (default 5)--max-chars NChunk size (default 8000)--header-lines "a" "b"Manually specify header/footer lines to remove

Workflow#

1. Download PDF (if URL)#

python

import urllib.request
urllib.request.urlretrieve(url, "report.pdf")

2. Extract#

Run from this skill directory:

bash

cd <skill-directory>
python scripts/extract_pdf.py /path/to/report.pdf [options]

3. Read & Answer#

Read the output .md file and answer based on the extracted content.

4. Clean Up#

Delete temporary PDF and .md files when done.

Limitations#

• Scanned/image-based PDFs: Cannot extract text without OCR. Install OCR dependencies and configure an API key.
• Embedded charts/graphs: Only text labels extracted, not chart data.
• Multi-column layouts: Use --layout flag for improved reading order via x_tolerance.
• TOC detection: Robust multi-format matching with validation. Very non-standard layouts may still require manual extraction.
• Diff: Uses text similarity (Jaccard on normalized lines), not page numbers. Threshold adjustable via --diff-threshold N (default 0.8).

Troubleshooting#

OCR fails with "No API key"

→ Set OCR_API_KEY environment variable or fill config.json.

OCR model rejects images

→ The configured model may not support vision; either choose a vision-capable model (e.g., qwen/qwen3.6-plus:free, stepfun/step-3.5-flash:free) or let the script auto-fallback by removing the model setting.

Rate limit errors

→ Reduce concurrent calls, switch to a paid tier, or try a different model provider.