Before you start
- Verified Search Console property (optional but makes debugging easier).
Fetch https://your-domain/robots.txt
Read the file line by line. User-agent: * blocks apply to all crawlers unless overridden by specific agents.
If you see Disallow: /wp-content/ or similar broad blocks, reconsider — modern themes need CSS and JS under wp-content to render properly for users and for Google’s renderer.
Align robots with your SEO plugin
Yoast and Rank Math can serve “virtual” robots rules — compare what the plugin thinks with what robots.txt returns in the browser.
After changes, use URL Inspection → View tested page → Screenshot to confirm resources are not blocked.
Use noindex on pages that should never rank
Thin thank-you pages, internal search results, and cart flows often should carry noindex via meta robots or X-Robots-Tag — prefer that over weird Disallow hacks when possible.
Request indexing only where it makes sense
Bulk “fetch everything” behaviour wastes crawl budget on large sites.
Prefer strong internal linking plus a clean sitemap over hundreds of manual inspection requests.
Structured data: accurate or skip it
FAQ and HowTo schema must match visible content on the page.
Spammy or misleading structured data can trigger manual actions — add schema only when truthful.