Design, implement, review, and harden news/information crawlers as production data pipelines. Use when the user asks to build, analyze, debug, or refactor news website crawlers, market news crawlers, public opinion/news scraping, article list/detail extraction, pagination, incremental backfill, anti-bot analysis, or crawler database schemas.
Design, implement, review, and harden news/information crawlers as production data pipelines. Use when the user asks to build, analyze, debug, or refactor news website crawlers, market news crawlers, public opinion/news scraping, article list/detail extraction, pagination, incremental backfill, anti-bot analysis, or crawler database schemas.