SEO Technical: Indexing#

Guides indexing troubleshooting and fix actions. For how to find and diagnose issues in GSC, see google-search-console.

When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)#

• Fix actions: noindex, canonical, content quality, URL Inspection; verify robots.txt does not block (see robots-txt)
• Noindex: Page-level index control; which pages to exclude and how. Complements robots-txt (path-level crawl control) and google-search-console (Coverage diagnosis)

Initial Assessment#

Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.

Identify issue from GSC (see google-search-console for Coverage report, issue types, diagnosis workflow). Then apply fix below.

Crawled - Currently Not Indexed#

CauseActionLow quality, duplicate, off-topicImprove content, fix duplicates, set correct canonicalStatic assets (CSS/JS)See belowFeed, share URLs with paramsUsually OK to ignore; or noindex, canonical to main URLImportant content pagesUse URL Inspection, verify canonical/internal links/sitemap, Request indexing

Static Assets (Next.js / Vercel)#

Vercel adds unique dpl= params to static assets per deploy, creating many "Crawled - currently not indexed" URLs.

DoDon'tKeep robots.txt allowing /_next/Do not block /_next/ (breaks CSS/JS loading). See robots-txtAccept static assets in GSC as expectedDo not block /_next/static/css/ or ?dpl=Use X-Robots-Tag for static assetsCSS/JS should not be indexed; no SEO impact

Static assets in "Crawled - currently not indexed" is normal and expected.

Other Issue Types (from GSC Coverage)#

IssueFixExcluded by «noindex» tagRemove noindex if accidental; keep if intentionalBlocked by robots.txtSee robots-txt; remove Disallow for important pathsRedirect / 404Fix URL or add redirectDuplicate / CanonicalSet correct canonical; usually OKSoft-404Page returns 200 but content says "not found" or empty—Google may treat as 404. Fix: return 404 status for truly missing pages; or add real content for 200 pages

Soft-404#

A soft-404 occurs when a page returns HTTP 200 but the content indicates the page doesn't exist (e.g. "Page not found" message, empty state). Google may treat it as 404 and exclude from index.

FixWhenReturn 404Page truly doesn't exist; use proper 404 statusAdd contentPage is intentional (e.g. empty search results); ensure substantive content or use noindexRedirectIf URL moved, use 301 to correct destination

Noindex Usage#

• How: metadata.robots = { index: false } or <meta name="robots" content="noindex"> or X-Robots-Tag
• Rationale: Not all site content should be indexed; noindex is a valid choice for many pages
• Caution: Avoid noindex on important content pages
• With robots.txt: robots.txt = path-level crawl control; noindex = page-level index control. Do not block noindex pages in robots.txt—crawlers must access the page to read the directive. Use both: robots for /admin/, /api/; noindex for /login/, /thank-you/, etc. See robots-txt for when to use which.
• nofollow ≠ noindex: nofollow controls link equity only; it does not prevent indexing. To exclude from search, use noindex. See page-metadata for meta robots implementation.

Page Types That Typically Need Noindex#

CategoryPage TypesTypical MetaReasonAuth & AccountLogin, Signup, Password reset, Account dashboardLogin: noindex,nofollow; Signup: noindex,followNo search value; login indexed = security risk; signup follow allows crawl of Privacy/Terms linksAdmin & PrivateAdmin, Staging, Test pages, Internal toolsnoindex,nofollowNot for public; avoid discoveryConversion EndpointsThank-you, Confirmation, Checkout success, Download gatenoindex,followPost-conversion; no SERP value; allow link equitySystem & Utility404, Internal search results, Faceted/filter URLsnoindex,follow or noindex,nofollowThin/duplicate; 404 = error stateLegalPrivacy, Terms, Cookie Policy (optional)Often noindex,followLow-value indexed; reduces clutterDuplicate & ThinPrinter-friendly, Parameter URLs, Near-duplicatenoindex,follow or canonicalDuplicate content; canonical preferred when possibleLow-ValueMedia kit, Feedback board (external), Thin pressnoindex or index for brand queriesCase-by-case

noindex,follow vs noindex,nofollow: Use noindex,follow for most cases—excludes from SERP but allows link equity. Use noindex,nofollow only for login (security), staging, or temporary test pages.

Google Indexing API#

TypeTypical useJobPostingJob boardsBroadcastEventLive platforms

Requirements: Enable Indexing API, create service account, add owner in Search Console, request quota (default 200 URLs/day).

Output Format#

• Action items: Prioritized fixes
• References: Page indexing report

Related Skills#

• google-search-console: Find and diagnose indexing issues in GSC
• robots-txt: Path-level crawl control; when to use robots.txt vs noindex; do not block /_next/ or noindex pages
• page-metadata: Meta robots implementation; noindex vs nofollow
• xml-sitemap: Submit and maintain sitemap
• indexnow: Faster indexing for Bing
• canonical-tag: Resolve duplicate content