Web crawler: what it means and how it works

Why it matters

A page can only appear in a search engine’s results after that engine’s crawler has fetched it. If a crawler cannot reach a page, because no link points to it, robots.txt blocks it or the server returns errors, nothing later in the pipeline can fix that.

Crawlers are also a cost and a risk. Every request uses server capacity, and some traffic that claims to be Googlebot is not. Knowing which crawlers visit, how often and what they receive is basic operational knowledge for any large site.

How it works

A crawler keeps a queue of URLs. It takes a URL, checks the host’s robots.txt, requests the page, records the HTTP status and the response, extracts links and adds new URLs to the queue. Scheduling decides what to fetch next and how fast, based on how important and how changeable a page seems and how much load the server can take.

Each crawler identifies itself with a user-agent string. That string is only a claim: anyone can send Googlebot in a request. Search engines publish ways to verify their crawlers, such as reverse DNS lookups or published IP ranges.

Example

A car marketplace publishes 400 new listings a day. Googlebot finds them through category pages and the XML sitemap. Server logs show Googlebot spending most of its requests on sorted and filtered versions of the same search pages, while new listings wait days for a first visit. The crawler is working as designed; the site is giving it too many low-value URLs to choose from.

How to check crawler activity

1. Open the Crawl stats report in Google Search Console for Googlebot’s request volume, response codes and file types.
2. Read server or CDN access logs to see every crawler, not just Google’s: requests by user agent, status code and URL pattern.
3. Verify claimed search engine bots before trusting the numbers. Unverified traffic using a Googlebot user agent should be treated as unknown.
4. Check robots.txt rules for each crawler you care about, including AI crawlers, and confirm the rules do what you intend.

Common mistakes and misconceptions

• Myth If a page is crawled, it will be indexed.

  In fact Crawling only fetches the page. The search engine still decides whether to index it, and many crawled pages are not indexed.

• Myth A Googlebot user agent in the logs means Google visited.

  In fact User agents are easy to fake. Only verified requests (reverse DNS or published IP ranges) prove the visitor was Google.

• Myth Crawling more often means ranking higher.

  In fact Crawl frequency reflects how often a crawler expects a page to change and how important it seems. It is not a ranking signal you can raise directly.

Related concepts

• Crawling, rendering & indexingCrawl budgetHow many URLs a search engine is able and willing to crawl on a site; a concern mainly for large sites.Read the entry
• Crawling, rendering & indexingRobots.txtThe file at /robots.txt that tells crawlers which URLs not to fetch; it controls crawling, not indexing.Read the entry
• SEO auditing & troubleshootingCrawl logsServer or CDN request records filtered to bots: what crawlers really fetched and received.Read the entry
• Crawling, rendering & indexingRenderingRunning a page’s code to build the final content a search engine indexes.Read the entry
• Crawling, rendering & indexingIndexingWhen a search engine stores a page so it can be shown in results; crawling does not guarantee it.Read the entry
• AI search & generative searchAI crawlersBots that fetch pages for AI training, AI search indexes or live user requests.Read the entry

Check it on your site

Keep learning