Robots.txt

What robots.txt does

Located at yoursite.com/robots.txt, this file tells search engine crawlers which parts of your site they can access. It can block crawling of admin sections, duplicate content, or resource-heavy areas that waste crawl budget.

Robots.txt is a suggestion, not enforcement. Well-behaved crawlers respect it, but malicious bots ignore it. Never rely on robots.txt for security—use proper authentication for sensitive content.

Common robots.txt uses

Blocking admin areas: Prevent crawlers from accessing /admin, /wp-admin, or dashboard sections. These provide no search value and waste crawl resources.

Managing duplicate content: Block parameter-based URLs, sorted/filtered views, or print versions if you can't use canonical tags.

Protecting crawl budget: On large sites, blocking low-value sections ensures crawlers spend time on important pages. Small sites rarely need this—crawl budget isn't typically limiting.

Sitemap location: Robots.txt can specify your XML sitemap location, helping crawlers discover all pages.

Robots.txt vs meta robots

Robots.txt prevents crawling—search engines never fetch the page. Meta robots allows crawling but controls indexing. If you block a page with robots.txt, you can't add a noindex meta tag because crawlers never see it.

For pages you want removed from search results but need to pass link equity through, use meta robots noindex, not robots.txt. Blocked pages can't pass link equity.

Common robots.txt mistakes

Accidentally blocking important pages destroys traffic invisibly. A typo blocking /services instead of /admin hides your service pages from search. Test robots.txt changes carefully.

Blocking CSS or JavaScript prevents search engines from rendering pages properly, potentially affecting rankings. Modern crawlers need these resources to evaluate pages correctly.

Implementation approach

We configure thoughtful robots.txt files for every site: allowing full access to valuable content, blocking truly unnecessary sections, and specifying sitemap locations. This guides crawlers efficiently without accidentally blocking important pages.