AI SEARCH & GENERATIVE SEARCH
AI crawlers: what they are and how they work
Bots that fetch pages for AI training, AI search indexes or live user requests.
- Intermediate level
- 3 primary sources
- Reviewed 29 October 2026
- TopicAI search & generative searchAlso in Crawling, rendering & indexing, Technical SEO
- LevelIntermediateAssumes the basics of how search works
- Also calledAI bots
- Sources3 primary sourcesOpenAI, Google Search Central and more
- Reviewed29 October 2026Re-checked every 3 months
On this page
Quick definition
AI crawlers are bots run by AI companies to fetch web content, either to train models, to build an AI search index, or to fetch a page on demand when a user asks. Each purpose usually has its own user agent, so robots.txt can allow one and block another.
Crawlers by purpose
| Purpose | Examples | Blocking it means |
|---|---|---|
| Model training | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) | Content is not collected for training by that operator |
| AI search index | OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot | Pages may not appear as sources in that AI search product |
| User-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot open your page when a user asks it to |
| Control tokens (not crawlers) | Google-Extended, Applebot-Extended | Opts content out of some AI uses; the normal search crawler still fetches |
Operators document their user agents and how they treat robots.txt, and their policies change; some user-triggered fetchers do not follow robots.txt in the same way as crawlers. Check each operator’s current documentation, and verify bots by IP where operators publish ranges.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Why it matters
Blocking every AI bot keeps content out of training sets but can also remove a site from AI search answers and stop assistants opening its pages for users. Allowing everything has costs too: server load and content reuse. A deliberate policy per purpose is better than a blanket rule copied from elsewhere.
Many AI crawlers fetch HTML without running JavaScript, so content that depends on client-side rendering may be invisible to them even when Googlebot sees it.
How to check it
- Test your robots.txt against each AI user agent you care about.
- Look at server or CDN logs to see which AI crawlers actually visit, how often and what they receive.
- Check whether firewall or bot-protection rules block crawlers your robots.txt allows.
Common mistakes and misconceptions
Myth Blocking GPTBot removes you from ChatGPT answers.
In fact GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and user requests use ChatGPT-User.
Myth Google-Extended is a crawler.
In fact It is a robots.txt token. Googlebot does the fetching; the token controls some AI uses of what it fetched.
Check it on your site
Free tools, no sign-up
- FreeAI Crawler Access CheckerSee which AI crawlers a site’s robots.txt allows or blocks, from GPTBot and ClaudeBot to Google-Extended and PerplexityBot, with the deciding rule.Open the tool
- FreeAI Robots.txt Policy GeneratorChoose whether to allow AI training crawlers, AI search crawlers and user-initiated agents, and get robots.txt rules for every documented AI bot.Open the tool
- FreeAI Crawler Log AnalyzerPaste server access log lines and see which AI crawlers visited, which were real (verified by IP) and which only borrowed the name.Open the tool
In the NdovuSEOLabs platform
Run the same checks on every page of your site, with your own data.
Keep learning
Sources and review
- Overview of OpenAI crawlers OpenAI
- Overview of Google crawlers and fetchers Google Search Central
- RFC 9309: Robots Exclusion Protocol IETF
Written and maintained by Ndovu SEO Labs. First published 29 October 2026. Facts last checked against the sources above on 29 October 2026. How we write and review the glossary.