AI SEARCH & BOTS
AI Robots.txt Policy Generator
AI crawlers come in three kinds: ones that collect pages to train models, ones that fetch pages to answer searches with links, and agents that fetch a page because a person asked. Each has its own robots.txt token. Decide what you want for each kind and get correct rules for every documented bot.
- Runs in your browser
- Nothing you paste is uploaded
- Free, no sign-up
- CheckGroups for each documented AI token
- CheckExisting groups left untouched
- CheckAllow groups only when the * group blocks everything
- CheckEvery AI crawler's verdict in the new file
Free
no sign-up, no usage limit
0 uploads
what you paste stays on your device
4 checks
what the tool looks at
Offline
works once the page has loaded
Loading the tool…
HOW IT WORKS
How to use the AI Robots.txt Policy Generator
Your current robots.txt (optional) and a choice for each group of AI crawlers.
Paste your current robots.txt so nothing is lost.
Choose allow or block for training, AI search and user-initiated agents.
Copy the file, upload it to your site’s root and check it with the AI Crawler Access Checker.
WHAT IT CHECKS
What it looks at, and how to read it.
robots.txt with AI groups added, checked crawler by crawler, to copy or download.
What it checks
- Groups for each documented AI token
- Existing groups left untouched
- Allow groups only when the * group blocks everything
- Every AI crawler’s verdict in the new file
How to read the results
- Blocking training while allowing AI search is a common choice for publishers and shops.
- Allowed crawlers follow your * rules unless they need their own group.
The concepts behind it, in the SEO glossary: AI crawlers.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Blocking OAI-SearchBot when you want to appear in ChatGPT search.
- Blocking Google-Extended and expecting Googlebot to stop.
- Pasting a list from a blog that also blocks search engines.
A worked example
A shop blocks training crawlers and allows AI search: the generator adds Disallow groups for GPTBot, ClaudeBot, Google-Extended, CCBot and others, and leaves OAI-SearchBot and PerplexityBot on the site-wide rules.
Limitations
- robots.txt is a request; well-behaved crawlers follow it, others do not. It does not remove content already collected.
- Covers the AI crawlers in the NdovuSEOLabs bot registry, whose operators document a robots.txt token.
FAQ
Questions about the AI Robots.txt Policy Generator
Straight answers about what it does and where its limits are.
Will this remove my content from AI models?
No. It asks compliant crawlers not to fetch pages from now on. Content already collected is not deleted.
Does blocking AI crawlers affect Google rankings?
No. Googlebot and Google-Extended are separate: Google-Extended only controls use in Gemini and Vertex AI.
Need this for a whole website, over time?
The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and audits every page on a schedule.