Skip to main content
Request access

SEO AUDITING & TROUBLESHOOTING

Crawl logs: what they show and how to read them

Server or CDN request records filtered to bots: what crawlers really fetched and received.

Check it on your site
All glossary terms
  • Advanced level
  • 2 primary sources
  • Reviewed 29 October 2026
At a glanceSEO glossary
  • TopicSEO auditing & troubleshootingAlso in Crawling, rendering & indexing, AI search & generative search, Search analytics & measurement
  • LevelAdvancedFor technical SEOs and developers
  • Also calledserver access logs
  • Sources2 primary sourcesGoogle Search Central
  • Reviewed29 October 2026Re-checked every 12 months
On this page
  1. Quick definition
  2. Why it matters
  3. How it works
  4. What to look for
  5. Common mistakes
  6. Related concepts
  7. Check it on your site
  8. Sources and review

Quick definition

Crawl logs are the records your web server or CDN writes for every request, filtered to the requests made by crawlers. They are the only first-party record of what search and AI bots actually requested and what your server returned.

Why it matters

Search Console shows Google’s view, sampled and delayed, for Google only. Logs show every request from every crawler, as it happened: which URLs Googlebot spends time on, which AI crawlers visit, which bots are fake, and which responses were errors or redirects.

How it works

A typical access log line holds the time, client IP, request method and path, status code, response size, referrer and user agent. Log analysis groups these by crawler and URL pattern.

66.249.66.1 - - [21/Oct/2026:06:14:09 +0000] "GET /phones/redmi-note-14/ HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
One request: time, path, 200 status, size and the claimed user agent.
  • Verification: confirm claimed bots by reverse DNS or published IP ranges before counting them.
  • Sources: origin servers, CDNs such as Cloudflare, and hosting panels. Behind a CDN, cached hits may never reach the origin log.
  • Privacy: logs contain visitor IP addresses; keep only bot traffic where possible, and set retention limits.

What to look for

  1. Share of verified search engine requests by status code: rising 404, 5xx or 3xx shares deserve attention.
  2. Requests by URL pattern: are crawlers spending time on filters, parameters or old URLs?
  3. Time from publishing to first crawl for new pages.
  4. AI crawler activity per bot, compared with your robots.txt policy.
  5. Unverified traffic using search engine user agents, which may be scrapers.

Common mistakes and misconceptions

  • Myth Search Console’s Crawl stats replace log analysis.

    In fact Crawl stats cover Google only and summarise. Logs cover every crawler and every URL.

  • Myth Analytics tools show bot visits.

    In fact Most analytics tools run JavaScript in the browser and deliberately exclude bots. Crawlers rarely appear there.

Check it on your site

Free tools, no sign-up

In the NdovuSEOLabs platform

Run the same checks on every page of your site, with your own data.

Sources and review

Written and maintained by Ndovu SEO Labs. First published 29 October 2026. Facts last checked against the sources above on 29 October 2026. How we write and review the glossary.