Crawl budget: what it means and how it works

Why it matters

When a site generates far more URLs than a crawler will fetch, important pages wait. New stock, updated prices and new listings appear in search late, while crawlers spend requests on parameter variations, redirects and errors.

Google’s guidance says most sites do not need to think about it: the advice targets sites with roughly a million or more pages that change weekly, sites with ten thousand or more pages that change daily, and sites with many URLs reported as discovered but not indexed.

How it works

• Crawl capacity limit: rises when the server responds quickly and falls when it slows down or returns server errors.
• Crawl demand: driven by how popular and how stale URLs seem, and by events such as a site move.
• Waste: duplicate URLs, long redirect chains, soft 404s and infinite spaces (calendars, filters) consume requests that could go to useful pages.

Example

A classifieds site has 80,000 live ads, but its filters for price, location, sort order and page number combine into millions of crawlable URLs. Logs show 70% of Googlebot requests going to filter combinations. Blocking crawl of the low-value parameters in robots.txt and linking new ads from fresh category pages shifts requests back to the ads.

How to check it

1. Compare the number of URLs you want indexed with Googlebot’s daily requests in Search Console‘s Crawl stats.
2. In server logs, group verified Googlebot requests by URL pattern and status code to see where requests go.
3. Look for rising server response times or 5xx errors in Crawl stats; both reduce crawl capacity.
4. If most important pages are crawled within days of changing, there is no crawl budget problem to fix.

Common mistakes and misconceptions

• Myth Every site should optimise crawl budget.

  In fact For a site with a few thousand URLs, Google generally crawls everything it wants. Indexing problems there usually have other causes.

• Myth Noindex saves crawl budget.

  In fact A noindexed page still has to be crawled for the noindex to be seen. Blocking crawl in robots.txt is what reduces requests.

Related concepts

• Crawling, rendering & indexingWeb crawlerThe automated program that fetches pages and follows links for a search engine, AI service or tool.Read the entry
• E-commerce SEOFaceted navigationListing filters that can generate huge numbers of near-duplicate URLs.Read the entry
• SEO auditing & troubleshootingCrawl logsServer or CDN request records filtered to bots: what crawlers really fetched and received.Read the entry
• Crawling, rendering & indexingRobots.txtThe file at /robots.txt that tells crawlers which URLs not to fetch; it controls crawling, not indexing.Read the entry
• Technical SEOSoft 404A not-found or empty page that wrongly returns 200 OK.Read the entry
• Technical SEO301 redirectA permanent move from one URL to another; the standard way to change URLs without losing search visibility.Read the entry

Check it on your site