Crawl budget

How crawl budget works

Search engines allocate limited resources to each site. They won't crawl your entire site every day—instead, they prioritise based on perceived importance, freshness, and site quality. Crawl budget represents this allocation: how many pages get crawled and how often.

For small sites under 1,000 pages, crawl budget rarely limits indexing. Search engines will discover and index your content without issue. Larger sites or those with technical problems may encounter crawl budget constraints preventing important pages from being indexed promptly.

What wastes crawl budget

Duplicate content at multiple URLs forces crawlers to process identical pages repeatedly. Infinite URL parameters from filters or session IDs create endless variations. Broken links waste crawls on 404 errors. Redirect chains slow crawlers unnecessarily.

Low-quality pages, excessive pagination, and complex JavaScript rendering all reduce crawl efficiency. If crawlers spend their budget on these issues, they have less capacity for your valuable content.

Optimising crawl budget

Implement canonical tags to consolidate duplicate URLs. Use robots.txt to block crawlers from low-value sections—admin pages, search result pages, or customer account areas.

Fix broken links promptly. Ensure redirects point directly to final destinations, not through chains. Improve site speed—faster sites get crawled more efficiently.

Monitoring crawl activity

Google Search Console reports crawl stats—pages crawled per day, crawl errors, and resources that couldn't be loaded. Significant drops in crawl rate often indicate technical problems worth investigating.

For most SMB sites, simply maintaining clean architecture and fixing technical issues keeps crawl budget adequate. Focus on eliminating waste rather than obsessing over maximising crawls.