Skip to main content

Try Base free for 14 days, no card neededPay yearly and get 2 months free See pricing

Last updated Agent setup

Report guide

Read Crawl Analytics

Which search and AI crawlers read your site, what they miss and what goes wrong.

The question it answers

Which search and AI crawlers read my site, which pages do they miss, and what goes wrong for them?

Where it is

Sidebar → Site health → Crawl Analytics. It needs a connected log source; until the first batch arrives, the page shows a setup wizard.

Crawl Analytics Overview with the view list, period and source pickers, the coverage notice and the Start here questions.
Crawl Analytics on Exelic Gadgets. Figures are blurred here.

Verified, unverified and spoofed

Search and AI counts are verified requests: the bot’s IP matched its operator’s published ranges, its reverse DNS checked out, or Cloudflare verified it. Other views also show unverified and spoofed requests, labelled as such.

Probable
Matched an older copy of the operator’s IP list.
Unverified
The operator publishes no way to check.
Spoofed
Came from a network the operator doesn’t use.

The views

Overview
“Start here” answers five questions (can the figures be trusted, is Googlebot getting in, where is the search crawl going, are AI crawlers getting in, is anything exposed), plus pages requested, search and AI requests, crawl issues and coverage by section.
Crawl issues
Problems in the last 7–30 days of logs, each with what to do, why it matters, how to check the fix and the affected URLs. Needs a week of data.
Errors & blocks
Responses by bot: not found includes 410; refused means 403 or 429.
Search crawling
Googlebot, Bingbot and others, their responses, and crawl ÷ clicks by section: above 3 means Google crawls a section far more than it earns.
Sitemap coverage
Each known page Recently observed, Stale (older than the stale threshold), Not observed, Excluded intentionally or Unknown.
AI crawling
AI search crawlers, training crawlers and user-directed fetchers, verified and refused. A request is not a citation.
Pages
All URLs with last crawl, response, whether they can rank, search demand and audit issues; site sections.
Activity
Traffic, a sample of requests, every crawler and automated traffic.

Filters

Period (Last 24 hours to Last 90 days or custom), and the data source when you have more than one. Requests can be filtered by path, verification, response and crawler.

Coverage notice

Every report says which source it read and whether it covers the whole period (Collecting since …: part of this period has no data). Open What this data includes: some sources can’t see cache hits or edge blocks.

Alerts

Checked every hour against each source’s own history, after one to two weeks of baseline: a source stopped sending data, robots.txt changed, a new crawler, a crawler stopped, errors spiked, impersonation, unusual volume. They appear in Site Monitoring → Alerts.

When data is empty, late or unavailable

No verified search or AI crawler requests
Nothing verified in this period; check the source covers your pages.
Issues need a week of data
Wait a week after connecting.
Waiting for first batch
See server logs troubleshooting.

Next step

Was this helpful?