AI SEARCH & BOTS
AI Crawler Access Checker
AI companies crawl for different reasons: to train models, to answer searches with links back, or to fetch a page someone asked about. Each has its own robots.txt token, so blocking one does not block the others. This check reads a site’s robots.txt and gives the verdict for every documented AI crawler.
- Checks the live page
- Public addresses only
- Free, no sign-up
- Checkrobots.txt state: present, missing (4xx) or unreachable (5xx)
- CheckVerdict for each AI crawler on the homepage or the path you enter
- CheckThe group and rule that decide each verdict
- CheckTraining, AI search and user-initiated crawlers told apart
Free
no sign-up
Live
fetched when you run the check
4 checks
what the tool looks at
30 a minute
checks per visitor, to keep it free
Loading the tool…
HOW IT WORKS
How to use the AI Crawler Access Checker
A domain or page URL.
Enter a domain, or a page URL to test a specific path.
Read the table: blocked or allowed, and the line in robots.txt that decides it.
Change robots.txt to match your policy, then check again.
WHAT IT CHECKS
What it looks at, and how to read it.
Every AI crawler in the NdovuSEOLabs bot registry with its purpose, robots.txt token, verdict for the path and the rule that decides it.
What it checks
- robots.txt state: present, missing (4xx) or unreachable (5xx)
- Verdict for each AI crawler on the homepage or the path you enter
- The group and rule that decide each verdict
- Training, AI search and user-initiated crawlers told apart
How to read the results
- Blocking training crawlers and allowing AI search crawlers is a common middle path.
- A crawler that follows the * group is affected by every site-wide rule.
- An unreachable robots.txt makes compliant crawlers stop crawling the whole site.
The concepts behind it, in the SEO glossary: Robots.txt, AI crawlers.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Blocking GPTBot and expecting ChatGPT search results to disappear: OAI-SearchBot is separate.
- Blocking Google-Extended and expecting Googlebot to stop: it only controls AI use.
- Copying a long block list that also blocks search crawlers you want.
A worked example
A news site wants to appear in AI answers but not in training data. The check shows GPTBot and ClaudeBot blocked, OAI-SearchBot and PerplexityBot allowed, as intended.
Limitations
- robots.txt is a request; it shows what a site asks for, not what crawlers do. Some agents fetch for a user and may not follow it.
- Covers the AI crawlers in the NdovuSEOLabs bot registry, whose operators document their robots.txt tokens.
FAQ
Questions about the AI Crawler Access Checker
Straight answers about what it does and where its limits are.
Does blocking AI crawlers remove my content from AI answers?
It stops compliant crawlers from fetching pages from now on. Content already collected, and copies on other sites, are not affected.
Which AI crawlers are included?
Those in the NdovuSEOLabs bot registry: crawlers whose operators publish their user agent and robots.txt token, such as OpenAI, Anthropic, Google, Apple, Perplexity, Meta and Common Crawl.
See which AI bots really visit, verified from your logs.
Free tools check one thing at a time. The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and keeps checking every page.