hfpclawer Paper Search & Download#

Part of the Exo suite — literature ( hfpclawer) → experiments ( expflow-pde) → proofs ( omega-architect). Three independent CLIs that meet through files and CLI calls, never imports. Entry skill: exo-suite-linkage (wiring, cost tiers low → medium → high, degradation ladder). Install: uv tool install hfpclawer · uv tool install expflow-pde · uv tool install "omega-architect @ git+https://github.com/diamond2nv/omega-architect@v0.2.3"

🔒 Sanitization: This skill ships in the public repo. Never embed private LAN IPs, real person names, or machine codenames in examples — use <placeholder> / Jane Doe / dev@example.com. Real values live in the repo's gitignored .hermes/internal-guide.md. See repo AGENTS.md "Public-Release Sanitization".

A multi-source academic paper pipe: search across arXiv / HuggingFace Papers / OpenReview / PapersWithCode, deduplicate by title, download PDFs, convert to Markdown, and optionally sync to a wiki.

Who this is for: Researchers who want a daily "new papers on my topic" feed without manually checking multiple websites.

Overview#

Typical workflow in one command:

text

hfpclawer search           # Discover new papers across sources
   └── ranked by relevance to your keywords
hfpclawer download         # Download PDFs for matched papers
   └── 8 concurrent streams
hfpclawer convert --to-wiki # PDF → readable Markdown + wiki sync

Direct fetch of a known arXiv ID (v0.16.11+)#

hfpclawer fetch <arxiv_id> downloads one paper by ID with a built-in network escape ladder — TCP quick-probe → QUIC/HTTP-3 → browser-hint. Use it whenever arXiv TCP/443 is RST-reset (common on CN networks); QUIC over UDP/443 survives the reset and is the primary escape channel.

bash

hfpclawer fetch 2609.02737                    # PDF, auto transport
hfpclawer fetch 2608.06013 -k source -t quic  # TeX bundle, force QUIC

Robustness built in: bounded-memory streaming (512KB flush to .part), resume on interruption (fresh .part continues via Range; >24h stale .part reclaimed), and a sha256 integrity anchor printed on every completed fetch and appended to data/download_audit.jsonl.

Channel-verification trick: fetch the same arXiv ID twice — once from the official arXiv endpoint and once from a mirror (e.g. AlphaXiv; its real PDFs live at pdfs.assets.alphaxiv.org) — and compare sha256. Byte-identical hashes verify the QUIC channel end-to-end (cross-channel MITM detection).

Or run the full pipeline at once:

bash

hfpclawer full --max-pages 3 --to-wiki

Multi-Source Registry (v0.18+)#

Beyond the default HF/arXiv path, three biomedical adapters are registered — europepmc, biorxiv, medrxiv. They run only when enabled:

yaml

search:
  enabled: [hf_cli, arxiv_api, europepmc, biorxiv, medrxiv]

bash

hfpclawer source-list                               # which adapters exist
hfpclawer source-search europepmc "CRISPR screen"   # query one adapter directly

• Registration is not enablement — an adapter in the registry does nothing until its key is in search.enabled, so adding one cannot change existing behaviour.
• Keep personal query lists and real names in the gitignored config.local.yaml; search.biomed_queries: [] in the tracked file is the declared placeholder slot.
• Europe PMC returns no abstract unless resultType=core is set (the adapter sets it); bioRxiv/medRxiv expose a date-range API only, so keyword filtering happens at ingest.
• Transient failures retry through anti_crawl.max_retries / retry_http_codes / retry_delay_base.

Prerequisites#

Python ≥ 3.10. Pick the install that matches how you work — uv is recommended, because the CLI then lives in its own environment (no conflicts with your project's dependencies):

bash

# 1) Recommended — uv tool: isolated CLI install, `hfpclawer` on your PATH
uv tool install hfpclawer
hfpclawer init                    # writes config.yaml

# 2) Try it without installing anything (ephemeral, one-off runs)
# pin the version — `uvx`/`uv tool run` reuse an installed tool env (may run an older
# release), and an unpinned launch is a supply-chain (rug-pull) risk
uvx "hfpclawer==0.19.0" --help

# 3) Inside an existing project / venv (uv-managed)
uv pip install hfpclawer

# 4) No uv yet — pip and pipx both work
pip install hfpclawer             # or: python -m pip install hfpclawer
pipx install hfpclawer            # CLI-style install, functionally like `uv tool`

Optional extras — the core install stays deliberately small:

ExtraAddsInstallzotero``pyzotero — the Zotero read / write / ingest pathsuv tool install "hfpclawer[zotero]"``nlpspaCy pipeline for entity enrichmentuv tool install "hfpclawer[nlp]"``graphnetworkx + geopy for the citation graphuv tool install "hfpclawer[graph]"``llmlitellm for opt-in LLM helpers ( sniff)uv tool install "hfpclawer[llm]"

⚠️ nlp extra + PyPI (checked against the published 0.19.0 metadata): PyPI strips the direct-URL spaCy model, so that extra installs spacy only — fetch the model yourself with python -m spacy download en_core_web_sm. The loader falls back configured → en_core_web_md → en_core_web_sm and logs one actionable hint when none is present: entity enrichment degrades, nothing else breaks.

Where to find it: repo https://github.com/diamond2nv/hfpapers-crawler · PyPI https://pypi.org/project/hfpclawer/ · registry: clawhub inspect <slug>

Edit config.yaml with your search interests (see Configuration section below).

Quick Start

1. First-time Setup#

bash

# Create default config
hfpclawer init

# Edit the config to match your research interests
vim config.yaml
# → Change: search.queries, keywords.include_high, keywords.exclude

2. One-Shot Full Pipeline (daily use)#

bash

# Discover → Download → Convert → Wiki sync in one command
hfpclawer full

# Limit pages for a quick check
hfpclawer full --max-pages 3 --to-wiki

3. Step-by-Step (for debugging)#

bash

# Step 1: Search across all sources
hfpclawer search --max-pages 5

# Step 2: Download PDFs for matched papers
hfpclawer download

# Step 3: Convert PDFs to Markdown
hfpclawer convert

# Step 4: Sync to wiki directory
hfpclawer convert --to-wiki

4. Monitor New Papers Regularly#

bash

# Check what papers have been downloaded
hfpclawer list

# Show paper store statistics
hfpclawer store stats

# Start the real-time download monitor
hfpclawer monitor start

Configuration#

The config file config.yaml controls what papers are searched and downloaded:

yaml

search:
  max_per_dim: 50           # Papers per search query per source
  queries:
    - query: "neural operator"
      category: neural-operator
    - query: "physics-informed"
      category: physics-informed
    - query: "PDE solver deep learning"
      category: pde-solver

keywords:
  include_high:              # Papers must match these (OR)
    - "neural operator"
    - "pde"
    - "deep learning"
  include_low:               # Optional bonus keywords
    - "fourier"
    - "self-attention"
  exclude:                   # Exclude these topics
    - "quantum"
    - "llm"

classification:
  threshold_pass: 30         # Relevance score threshold (0-100)
  title_similarity_min: 0.40 # Dedup threshold

paths:
  data_dir: "data"           # SQLite DB location
  pdf_dir: "pdfs"            # Downloaded PDFs
  md_dir: "mds"              # Converted Markdown files

Available Commands#

CommandPurposeCommon Flagshfpclawer searchDiscover new papers--max-pages, --dry-run``hfpclawer downloadDownload PDFs(runs from search results)hfpclawer convertConvert PDF → MD--to-wiki syncs to raw/papers/``hfpclawer fullAll-in-one pipeline--max-pages, --to-wiki``hfpclawer listList downloaded papershfpclawer store statsPaper store statisticshfpclawer store exportExport store as JSON/CSV--format json``hfpclawer store verifyCross-verify paper metadata--arxiv-id``hfpclawer configShow current confighfpclawer mcpStart MCP server(for LLM integration)hfpclawer monitorDownload daemon controlstart, stop, status``hfpclawer dedupShow dedup statistics

Daily Routine Examples#

Morning — Check What's New#

bash

# Quick scan (3 pages per query, ~50 papers)
hfpclawer search --max-pages 3

# View results
hfpclawer store stats

Afternoon — Download & Read#

bash

# Download all new papers
hfpclawer download

# Convert to readable markdown
hfpclawer convert

# Read the best one
cat mds/2010.08895.md | head -80

Weekly — Full Pipeline#

bash

# Full sweep with wiki sync
hfpclawer full --max-pages 10 --to-wiki

# Validate references in newly added papers
hfpclawer audit verify "Key cited paper" --source openalex

Data Storage#

hfpclawer uses three tiers:

StorageLocationContentPersistenceSQLitedata/papers.dbMetadata, dedup, cross-refPersistentPDFspdfs/Raw paper PDFsDownload once, keepMarkdownmds/Converted textRegeneratable from PDFs

The paper store tracks:

• arXiv ID, title, authors, abstract
• Source of discovery (HF / arXiv / OpenReview)
• Download status, conversion status
• Wikified path (if synced)
• Cross-verification with Crossref (DOI validation)

Search · Breadth · Recommend (搜广推, v0.18+)#

Three families, each usable on its own — combined they form the discovery loop:

LayerCommandsWhat it does搜 Searchsearch, convert-tex, sniff, fetchmulti-source discovery (HF Papers + arXiv + OpenReview + biomedical), TeX source → Markdown with formulas preserved, opt-in LLM abstract triage, CN-aware single-paper transport广 Breadthgraph ( expand-hub --audit), pool, dedup, batchcitation-graph hub expansion with an audit trail of what was adopted vs truncated, positive-example pool, dedup statistics, queue-based batch download推 Recommendrecommend, profile, rankscored recommendations that show why (layer + query per row), interest profile, opt-in learned re-rankingbash

# Recommend papers for this repo (fuses config queries + repo/user profile, keeps suspect out)
hfpclawer recommend --limit 10
hfpclawer recommend --path ~/my-project        # read the profile from another repo

# Show the profile that drives it (repo block + private user profile)
hfpclawer profile            # repo: hfpclawer: YAML block in the nearest AGENTS.md
hfpclawer profile --user     # private ~/.hfpclawer/profile.yaml

# Grow the positive-example pool, then train the learned re-ranker (local lightgbm)
hfpclawer pool sync-favorited                  # Zotero favorites → positives
hfpclawer pool ingest-verified                 # human-approved papers
hfpclawer pool export --out data/train.jsonl
hfpclawer rank train --audit data/expand-audit.jsonl --out data/rank_model.txt

Why it is auditable: every recommendation row carries its provenance (which layer, which query, verification state). Suspect papers are gated out; verified ones are preferred.

Zotero Integration (read · write · ingest)#

Talks to Zotero Desktop's local server on port 23119 — the local server needs no cloud account; the zotero pyzotero extra is still required for these paths. Zotero must be running.

DirectionCommandsEndpointReadzotero check, list, search, get, tags, childrenlocal API /api/Writezotero push, zotero push-batchConnector protocol /connector/Ingestzotero ingestZotero local PDF → paper_store + wiki/raw + annotationsLearnpool sync-favoritedZotero favorites feed the recommendation poolbash

hfpclawer zotero check                        # verify connectivity first
hfpclawer zotero list --limit 10 --tag hfpclawer
hfpclawer zotero search "neural operator" --limit 5
hfpclawer zotero get ABC123
hfpclawer zotero push <arxiv-id>              # paper_store → Zotero
hfpclawer zotero ingest ABC123                # Zotero PDF → store + wiki + annotations

Override ZOTERO_API_URL only if Zotero listens elsewhere. Real Zotero user ids are private — keep them in gitignored config, never in tracked files.

Hermes Agent Environment#

hfpclawer is built to run inside Hermes Agent (and OpenCode) as a first-class tool — the agent discovers the skill, and every command below is callable without leaving the session.

1. Install the skill — place this folder under ~/.hermes/skills/research/<slug>/ (or install from ClawHub: clawhub inspect <slug> --file SKILL.md); Hermes loads it automatically and skill_view(name='<slug>') returns this file.

2. Register the MCP server so the agent calls the CLI as tools:

yaml

# ~/.hermes/config.yaml
mcp:
  servers:
    hfpclawer:
      command: "hfpclawer"
      args: ["mcp"]        # stdio mode — Hermes native MCP client

For OpenCode / debugging use HTTP mode: hfpclawer mcp --mode http --port 8765.

3. Environment variables — every one is optional; the pipeline runs with none set:

VariablePurposeHFPAPERS_DATA_DIRstate/DB root (XDG ~/.local/share/hfpclawer when installed; the checkout when run from source)HFPAPERS_CONFIG / HFPAPERS_LOCAL_CONFIGconfig file + private overlayS2_API_KEYSemantic Scholar — 10x faster (anonymous tier works)OPENALEX_POLITE_EMAILOpenAlex polite pool — 10x fasterZOTERO_API_URLZotero local API base (default http://127.0.0.1:23119/api/)HFPCLAWER_PEER_REPOS / HFPCLAWER_REPO_MAPaddress private sibling repos by tag instead of hard-coded paths

4. Cost profile — the mechanical layer (search / dedup / verify / audit / Zotero) needs low-cost by design — not zero-cost. The core path (search / dedup / verify / audit / Zotero) issues no LLM call and needs no API key, so the marginal cost per run is small; it is still not zero — bandwidth, disk and CPU are spent, and rate-limited upstreams (arXiv / OpenAlex / Semantic Scholar) can throttle or expect a key at volume. LLM features ( sniff, abstract triage) are opt-in and metered where they run; rank trains locally with lightgbm. Describe the cost as low, and keep the LLM steps explicit — do not advertise the stack as zero-token.

5. Keep private data private — real author lists, ORCIDs and Zotero ids belong in config.local.yaml (gitignored) or ~/.hfpclawer/profile.yaml, never in tracked files. The repo-side profile (the hfpclawer: block in a project AGENTS.md) is deliberately public-safe: neutral academic keywords only.

Common Pitfalls#

1. pip install needs to be in the right venv. If hfpclawer command is not found, check the active Python environment.
2. HuggingFace CLI rate limits. Too many queries per minute will trigger 429s. Reduce max_per_dim to 10 if this happens.
3. Scrapy spiders need scrapy extra installed. If you see ModuleNotFoundError: scrapy, run pip install hfpclawer[scrapy].
4. PDF conversion needs pymupdf4llm. Run pip install hfpclawer[pdf] if hfpclawer convert complains about missing pymupdf4llm.
5. Wiki sync defaults to raw/papers/. If you do not have a wiki directory, skip --to-wiki and read from mds/ directly.
6. First run creates a config.yaml. Edit it before running hfpclawer full, otherwise the default queries may not match your research area.

Verification Checklist#

• hfpclawer init creates a valid config.yaml
• hfpclawer search --dry-run validates config without network calls
• hfpclawer search --max-pages 3 returns real papers
• hfpclawer download downloads PDFs correctly
• hfpclawer convert produces readable Markdown
• hfpclawer store stats shows non-zero counts
• hfpclawer store verify --arxiv-id 2010.08895 cross-checks via Crossref