Meta robots

Common meta robots directives

noindex: Prevents search engines from adding the page to their index. Useful for thank-you pages, internal tools, staging environments, or duplicate content you can't consolidate. Page remains accessible but won't appear in search results.

nofollow: Instructs crawlers not to follow links on this page. Prevents link equity from passing to linked pages. Rarely needed—use nofollow attributes on specific links instead.

noarchive: Prevents search engines from storing cached versions of the page. Useful for pages with frequently-changing content or pricing that shouldn't be accessible through cached views.

nosnippet: Blocks search engines from showing text snippets or previews in search results. Only your title appears. This limits visibility, so use sparingly.

When to use meta robots

Thank-you pages after form submissions—no value to searchers. Staging or development sites you're testing publicly. Internal dashboards or account pages accessible to customers but not meant for search discovery.

Duplicate content you can't eliminate through canonical tags or redirects. Print versions of articles, filtered/sorted product views, or calendar pages with minimal unique content.

Meta robots vs robots.txt

Robots.txt blocks crawling entirely—search engines never fetch the page. Meta robots allows crawling but controls what happens with the content. If you need to prevent indexing but allow link equity to pass through the page, use meta robots noindex, not robots.txt.

Robots.txt directives are public—anyone can view your robots.txt file. Meta robots tags are visible in page source but require accessing the page first. Neither provides security—use proper authentication for sensitive content.

Implementation best practices

Use noindex sparingly. Most pages should be indexable unless they genuinely provide no search value. Accidentally noindexing important pages invisibly destroys traffic—QA carefully.

Monitor Google Search Console for pages excluded due to noindex tags. Ensure these are intentional, not accidental blocks on valuable content.