Web Crawling#

Use this skill when the user needs webpage retrieval rather than a search engine result. It fits crawling, scraping preparation, readable content extraction, and page structure inspection for a known URL.

When To Use It#

• The user already has a target webpage URL and wants its raw HTML, rendered HTML, or cleaned Markdown.
• The task is about content extraction, page inspection, scrape preparation, or converting a page into LLM-friendly text.
• The user can provide a direct url to crawl.
• The user needs page content from the source URL itself, not Google search results about that URL.

Representative Operations#

• html: Crawl Webpage (HTML) — Retrieve the raw HTML response for a page.
• renderedHtml: Crawl Webpage (Rendered HTML) — Retrieve DOM output after rendering for JavaScript-heavy pages.
• markdown: Crawl Webpage (Markdown) — Extract the main readable content as clean Markdown for summarization or downstream processing.

Request Pattern#

• 3 read-only GET operations are available in this skill.
• All operations require a direct url query parameter.
• No operation in this skill requires a request body.
• Choose renderedHtml for dynamic pages, html for raw source, and markdown for readable content extraction.

How To Work#

1. Read generated/operations.md before choosing an endpoint.
2. Start with one of these operations when it matches the user's request: html, renderedHtml, markdown.
3. Pick the smallest matching operation instead of guessing.
4. Ask the user for any missing required parameter. Do not invent values.
5. Call the helper with:

bash

node {baseDir}/bin/run.mjs --operation "<operation-id>" --api-key "$JUST_SERP_API_KEY" --params-json '{"key":"value"}'

Environment#

• Required: JUST_SERP_API_KEY
• This skill uses JUST_SERP_API_KEY only for authenticated Just Serp API requests.
• Keep JUST_SERP_API_KEY private. Do not paste it into chat messages, screenshots, or logs.
• Project site: Just Serp API.
• Authentication details: Just Serp API Docs.

Output Rules#

• Start with what was fetched: raw HTML, rendered HTML, or cleaned Markdown.
• Echo the target URL so the crawl scope is explicit.
• For markdown, surface the extracted readable content or key sections before raw JSON.
• For HTML-oriented requests, mention whether the user asked for source HTML or rendered output.
• If the backend errors, include the backend payload and the exact operation ID.