Website Scanner#

Comprehensive website analysis tool that scans domains and generates detailed technical reports.

When to Use#

Use this skill when you need to:

• Analyze a website's technical infrastructure (IP, DNS, WHOIS)
• Audit website content and SEO
• Extract structured data (Schema.org JSON-LD)
• Check robots.txt, llms.txt, sitemap.xml
• Perform deep scanning of multiple pages
• Generate comprehensive website reports

Prerequisites#

Required System Tools#

• dig - DNS lookup utility
• whois - WHOIS query tool

Python Dependencies#

bash

pip install requests beautifulsoup4 aiohttp

Optional (for deep scanning)#

• Playwright (for JavaScript-rendered pages)

bash

pip install playwright
playwright install chromium

Usage#

Basic Scan#

bash

python3 scripts/scan.py example.com

Deep Scan (follow sitemap)#

bash

python3 scripts/scan.py example.com --deep --max-pages 10

Save Reports#

bash

python3 scripts/scan.py example.com --output report.json --markdown report.md

Features#

1. Network Analysis#

• IP Resolution: IPv4 and IPv6 addresses
• Geolocation: Server location, ASN, organization
• DNS Records: A, AAAA, MX, NS, TXT, CNAME, SOA
• WHOIS Data: Registrar, creation/expiration dates, status

2. Content Analysis#

• Homepage Metadata: Title, description, viewport, charset
• Heading Structure: H1-H6 distribution
• Links: Internal vs external count
• Images: Alt text coverage
• Scripts/Styles: Resource count

3. Structured Data#

• Schema.org JSON-LD: Extract all structured data schemas
• Open Graph: Meta tag analysis
• Twitter Cards: Social media metadata

4. SEO Analysis#

• SEO Score: 0-100 rating
• Issues: Missing tags, optimization problems
• Recommendations: Actionable improvements

5. Crawler Files#

• robots.txt: Fetch and parse
• llms.txt: AI/LLM instructions
• sitemap.xml: Parse URL structure

6. Third-Party Data#

• Google Index: Check indexed pages (approximate)
• Server Detection: Technology stack identification

7. Deep Scanning#

• Follow sitemap URLs
• Analyze multiple pages
• Client-side rendering support (with Playwright)

Output Format#

Console Output#

Human-readable markdown report with:

• IP and network information
• DNS record summary
• WHOIS details
• Content statistics
• SEO score and issues
• Third-party metrics

JSON Export#

json

{
  "scan_info": {
    "url": "https://example.com",
    "domain": "example.com",
    "scan_date": "2026-04-02T10:00:00"
  },
  "ip_info": { ... },
  "dns_info": { ... },
  "whois_info": { ... },
  "content_analysis": { ... },
  "seo": { ... },
  ...
}

Markdown Export#

Formatted report suitable for documentation or sharing.

Examples#

Example 1: Quick Scan#

bash

python3 scripts/scan.py hugogu.cn

Output includes:

• IP addresses and location
• DNS configuration
• WHOIS registration info
• Homepage SEO analysis

Example 2: Full Audit#

bash

python3 scripts/scan.py hugogu.cn --deep --max-pages 20 \
  --output audit.json --markdown audit.md

Performs comprehensive analysis including:

• All network information
• Deep page crawling
• SEO audit
• Structured data extraction
• Full report export

Limitations#

1. Rate Limiting: Some queries (WHOIS, DNS) may be rate-limited
2. JavaScript Rendering: Basic scan doesn't execute JS (use --deep with Playwright)
3. Third-Party APIs: Traffic estimation requires paid API keys
4. Google Index: Approximate only (scrapes search results)

Troubleshooting#

"dig command not found"#

bash

# Ubuntu/Debian
sudo apt-get install dnsutils

# macOS
brew install bind

# CentOS/RHEL
sudo yum install bind-utils

"whois command not found"#

bash

# Ubuntu/Debian
sudo apt-get install whois

# macOS
brew install whois

DNS Resolution Fails#

Check if domain is accessible:

bash

nslookup example.com
dig example.com

Architecture#

text

┌─────────────────┐
│ WebsiteScanner  │
└────────┬────────┘
         │
    ┌────┴────┬──────────┬──────────┐
    │         │          │          │
┌───▼───┐ ┌──▼────┐ ┌───▼───┐ ┌───▼────┐
│ IP    │ │ DNS   │ │ WHOIS │ │ Content│
│Analyzer│ │Analyzer│ │Analyzer│ │Analyzer│
└───────┘ └───────┘ └───────┘ └───────┘
                                   │
                          ┌────────┼────────┐
                          │        │        │
                      ┌───▼───┐ ┌─▼────┐ ┌─▼────┐
                      │ SEO   │ │ JSON │ │ Deep │
                      │Analyzer│ │ -LD  │ │ Scan │
                      └───────┘ └──────┘ └──────┘

Integration with OpenClaw#

This skill can be invoked from OpenClaw to:

1. Scan competitor websites
2. Audit your own sites
3. Research domain infrastructure
4. Generate technical documentation

Example OpenClaw workflow:

text

User: "Scan hugogu.cn and tell me about its infrastructure"
→ Run scanner
→ Analyze results
→ Generate summary