UNDERSTAND · CRAWL ANALYTICS
Search & AI crawlers: see how Google, Bing and AI bots crawl your site.
See how search engines and AI crawlers access your website. Find pages they miss, identify crawl errors and understand crawling activity, from your own server and CDN logs. Every crawler is checked against its operator’s published IP ranges or DNS.
- Verified, not user-agent guesses
- Your own server and CDN logs
- No research credits
4 KPIs
pages requested, search and AI requests, crawl issues
Verified
IP ranges, reverse DNS or Cloudflare signal
6 views
overview, search engines, AI, pages, issues, activity
0 credits
built on your own logs
INSIDE THE PRODUCT
Are crawlers reaching your important pages?
Four numbers on the overview, then a view for each question: search engines, AI crawlers, pages and crawl issues.
Search engines
See what Google and Bing crawl, and what they miss
Requests and unique URLs from Googlebot, Bingbot and other verified search crawlers, the last observed crawl, how your server answered, crawling by site section and important URLs not observed during the period.
- Requests and unique URLs requested
- Responses: success, redirects, 404s, server errors
- Sitemap coverage and pages not observed
| Crawler | Unique URLs | Requests |
|---|---|---|
| Googlebot | 1,240 | 18,420 |
| Bingbot | 610 | 6,830 |
| Applebot | 140 | 820 |
| Other search crawlers | 95 | 410 |
AI crawlers
Know which AI crawlers really visit
OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, ChatGPT-User and more, verified and grouped by why they fetch. Each request counts in exactly one group, so the groups add up. A visit is not a citation; AI referrals and AI visibility are one click away.
- AI search, training and user-directed groups
- Unverified AI-related traffic kept apart
- Links to AI visibility and AI referrals
- AI searchOAI-SearchBot4,610 verified requests
- TrainingGPTBot3,450 verified requests
- User-directedChatGPT-User1,920 fetches
- Unverified“GPTBot” from 45.x.x.x380 requests, not counted
Pages
Every URL with its crawl evidence
An SEO URL explorer: page type, sitemap status, the last verified crawler visits, the response crawlers got, and audit issues where a site audit has run. Stores also get product, category, brand, variation and filter segments.
- Filter by page type, sitemap status and response
- Last Googlebot, Bingbot and AI crawler visit
- Product and category segments for stores
| URL | In sitemap | Last Googlebot visit |
|---|---|---|
| /phones/phone-a/ | Yes | 2 days ago |
| /phones/phone-b/ | Yes | Not observed |
| /phones/?brand=a&sort=price | No | 6 hours ago |
| /laptops/laptop-c/ | Yes | 11 days ago |
Crawl issues
Crawl issues in plain language, with the URLs behind them
“Search engines are hitting broken URLs”, “Googlebot is receiving server errors”, “AI search crawlers are being blocked”, “Crawlers keep requesting outdated redirects”. Each issue shows its evidence, affected URLs, source, confidence and limits, what to do, and how to check the fix worked.
- Affected URLs, not just a count
- Source and confidence on every issue
- Possible crawl inefficiency, never guessed
- ErrorSearch engines are hitting broken URLs23 URLs answer 404
- ErrorGooglebot is receiving server errors/checkout/ answered 503
- WarningAI search crawlers are being blocked403 to OAI-SearchBot
- WarningCrawlers keep requesting outdated redirects41 old category URLs
Activity
Every request, filtered to the question
Read individual crawler requests with filters for crawler, status, path, verification, source and date, and open the crawler directory entry for any bot.
- Filter by crawler, status and path
- Verified, probable, unverified or spoofed
- Source and date on every row
| Crawler | Path | Status |
|---|---|---|
| Googlebot | /phones/phone-a/ | 200 |
| OAI-SearchBot | /phones/budget/ | 200 |
| Bingbot | /old-category/ | 301 |
| “Googlebot” (spoofed) | /wp-login.php | 403 |
Crawl inefficiency
See where search crawlers spend requests
The share of search crawler requests that ended on a redirect, an error, a URL with parameters or a section usually kept out of search, with the URLs requested most. Shown as possible inefficiency, never as lost rankings.
- Redirects, errors and parameters
- The most-requested URLs behind each
- Some of it is normal, and says so
Robots.txt & AI
Check what your rules allow each AI crawler
Beside the AI crawler groups, see your robots.txt rules for each crawler and whether verified AI search crawlers were refused, with AI referral visits one click away.
- robots.txt per crawler
- Refused AI search requests
- AI referrals from GA4
- AllowedOAI-SearchBot4,610 requests answered 200
- AllowedClaude-SearchBot1,120 requests
- BlockedGPTBotDisallowed in robots.txt; 0 page requests
- RefusedPerplexityBot212 requests answered 403 at the server
WHAT IT MEASURES
What each crawl number counts.
What each metric tells you, and where it comes from.
| Metric | What it tells you | Source |
|---|---|---|
| Pages requested | Distinct URLs at least one verified crawler requested in the period. Requested is not indexed. | Your logs |
| Search crawler requests | Verified requests from Googlebot, Bingbot, Applebot, DuckDuckBot and other search crawlers. | Your logs, verified |
| AI crawler requests | Verified AI requests in three groups that add up: AI search, training and data, user-directed fetchers. | Your logs, verified |
| Crawl issues | Findings from the most recent 7–30 days of logs. | Your logs + Search Console + Site Audit |
| Not observed | Known pages no verified crawler requested, only when the source collected for the whole period. | Logs + known pages |
| Unverified AI-related traffic | Requests using an AI crawler’s name that did not verify; never in the totals. | Your logs |
| Responses | Success, redirects, 404s and server errors crawlers received. | Your logs |
UNDER THE HOOD
How Crawl Analytics verifies and counts crawlers.
How the numbers are produced, so you can judge them for yourself.
Verified by the operator’s own checks
A request that claims to be a crawler is checked against that operator’s published IP ranges, reverse and forward DNS, or Cloudflare’s verified-bot signal. It is then verified, probable, unverified or spoofed, and only verified requests reach the totals.
Groups that add up
Search and AI requests never overlap, and AI requests fall into exactly one of three groups: AI search, training and data, or user-directed fetchers. A fetch made for a person counts as user-directed even if the operator also runs AI search.
Known pages from your evidence
Pages are known from Search Console, your latest site audit and, for WooCommerce, the catalogue sync. A known page is “not observed” only when the source collected for the whole period and is still delivering.
Coverage stated every time
Each report names its source and whether it covered the whole period, and comparisons with the previous period appear only when that period was fully collected too.
HOW IT WORKS
From a connected log source to a verified fix.
Four steps from connecting your data to a decision you can stand behind.
Connect a source
Add the WordPress plugin, Cloudflare Logpush or Analytics, or a log file import under Settings → Data Sources.
Check the coverage
Read the one-line notice: which source, whether it covered the whole period, and what it cannot see.
Open an issue
Each issue lists the affected URLs, evidence, confidence and what to do.
Verify the fix
Fix the page or rule, then watch the next period’s requests and responses.
WITH YOUR AI AGENT
Ask what crawlers are doing on your site.
Your agent reads verified crawler activity and issues, and inspects single URLs across every module.
- Uses the same data as the web app
- Paid tools estimate their cost first
- Saves the report to the project
YouAre AI search crawlers reaching our new category pages?
get_request_intelligenceget_bot_activityget_url_intelligence
AgentOAI-SearchBot and Claude-SearchBot requested 9 of the 14 new category pages this week, all answered 200. PerplexityBot was refused with 403 on every request; your firewall rule is blocking it, not robots.txt.
YouWhich important pages has Googlebot not requested?
AgentEleven pages in your sitemap with Search Console impressions were not observed in the last 30 days, mostly deep product variations. The source covered the whole period, so this is a real gap.
Illustration · sample data
USE CASES
Built for the way you work.
How teams put it to work.
Check products get crawled
See which product, category and variation pages Googlebot has not been observed requesting, and the filter URLs it requests instead.
Check AI crawlers can reach you
Confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch your key pages and are not blocked.
Find crawl errors before rankings move
See Googlebot hitting 404s and server errors on pages with impressions, and fix them first.
Check a firewall change
Confirm a new WAF or bot rule did not start refusing verified search or AI search crawlers.
ALSO INCLUDED
More from Crawl Analytics.
More capabilities that come with it, in the same project.
Alerts
Crawl changes raise alerts in the platform’s shared alerts.
Inspect URL
One URL across logs, Search Console, analytics and AI.
Crawler directory
Every crawler, its operator, its robots.txt token and how it is verified.
DATA & LIMITS
Know what the numbers can and cannot tell you.
Where the data comes from, how fresh it is, and the questions it cannot answer on its own.
| Source | What it provides | Freshness & limits |
|---|---|---|
| WordPress Crawl Log plugin | Requests that reach WordPress, verified by IP ranges and DNS. | Cannot see CDN cache hits or edge blocks. |
| Cloudflare Logpush | Every request at the edge, including cache hits and blocks. | Verified by IP ranges, DNS and Cloudflare’s signal. |
| Cloudflare Analytics | Aggregated edge requests. | User agent only; no verification. |
| Server log file import | Your web server’s access logs. | Verified by IP ranges and DNS; origin only. |
Requested is not indexed
A crawler request proves a fetch. It does not prove indexing, an AI citation, training or a visit.
Not observed is not never
A page is “not observed” only when the source covered the whole period; otherwise the answer is “can’t tell yet”.
Your source sets the view
Origin collection cannot see requests a CDN served from cache or refused at the edge.
Inefficiency is not a penalty
Search engines do not trade redirect requests one-for-one against other pages, so it is shown as possible inefficiency.
WORKS WITH
Connected to the tools you already use.
The sources and clients this feature reads from or works alongside.
CHOOSE THE RIGHT VIEW
Which view answers which question.
Start where the question points. Each view keeps its own source.
| Your question | Start with | Why |
|---|---|---|
| Which crawlers request my pages? | Crawl Analytics | Verified requests from your logs. |
| What did Google index? | Indexing intelligence | Google’s own index status. |
| What does my site look like to a crawler? | Technical site audit | Ndovu’s own crawl of your links. |
| Do AI answers cite me? | AI search visibility | Tracked answers and citations. |
LEARN & TRY
Guides and free tools for the same job.
Read how the evidence works, or try a free check before you connect a project.
Crawler directory
Every crawler, its operator, its robots.txt token and how to verify it.
Robots.txt URL tester
Check whether a crawler may fetch a URL.
AI crawler log analyzer
Paste log lines and count crawler requests.
FAQ
Crawl Analytics: common questions.
Straight answers about what it does, what it costs and where its limits are.
Where does the data come from?
From your own logs: the WordPress crawl-log plugin or Cloudflare sources, connected under Settings → Data Sources. A setup wizard walks you through it, and every report shows its source and collection coverage.
Is this the same as the site audit crawler?
No. The site audit is Ndovu’s own crawler visiting your site. Crawl Analytics shows how Google, Bing and AI crawlers visit it, from your logs.
How are crawlers verified?
Against each operator’s documented check: published IP ranges, reverse and forward DNS, or Cloudflare’s verified-bot signal. Requests that only claim a crawler’s name are kept apart.
Does a crawler visit mean I will be cited?
No. A request proves the page was fetched. It does not prove indexing, an AI citation or a visit; those are shown separately.
What can the WordPress plugin not see?
It collects at your origin, so requests answered from a CDN cache or blocked at the edge never reach it. Cloudflare sources can add those.
Does it cost credits?
No. Crawl Analytics runs on your own logs and needs no DataForSEO credits.
Which crawlers are recognised?
Search engines such as Googlebot, Bingbot, Applebot and DuckDuckBot, and AI crawlers such as OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot and PerplexityBot, among others in the crawler directory.
Is my own site audit counted?
No. Ndovu’s audit crawler is never counted as crawler activity.
Can I import a log file?
Yes. Server log file import is one of the data sources, alongside the WordPress plugin and Cloudflare.
Does it compare with the previous period?
Only when the source collected for the whole previous period, so a new connection never looks like a sudden rise.
Does it block crawlers?
No. It reports what crawlers did. Blocking happens in your robots.txt, firewall or CDN.
What are the limits of the data?
Everything is observed evidence, not a verdict. A crawler request does not prove indexing, an AI citation or a visit. “Not observed during this period” is not “never crawled”, and origin-level collection such as the WordPress plugin cannot see CDN cache hits or edge blocks.
How do I get access?
Platform access is reviewed. Request access with your website and the data sources you use, and the team will help you connect them.
See it with your own data.
Tell us about your website. We will review access and help you connect the right sources.