AI BOTS · COMMON CRAWL
CCBot
An open web archive that many AI models are trained on; blocking it affects future datasets.
- Follows robots.txt
- Can be verified
- Docs checked 28 Oct 2026
- VerifiableIdentityCloudflare
- Obeysrobots.txtFollows robots.txt
- PurposeAI training, Data collectionFor an AI product
- TokenCCBotUse it in a User-agent line
robots.txt
The operator documents that this bot follows robots.txt rules.
Block CCBot
User-agent: CCBot
Disallow: /Allow CCBot
User-agent: CCBot
Allow: /Quick facts
- robots.txt token
CCBot- User agent contains
- CCBot
- Operator
- Common Crawl
- Category
- AI bots
- Purpose
- AI training, Data collection
- robots.txt
- Follows robots.txt
- Verification
- Can be verified
- Documentation
- commoncrawl.org · checked 28 Oct 2026
If you block it
Stops this crawler collecting your pages for model training from now on. It does not change your search rankings.
Check a visit is real
- Listed by Cloudflare as a verified bot
Where this comes from
The same registry NdovuSEOLabs uses to classify crawler traffic in your server and CDN logs, built from the operator’s own documentation.
Registry version 2026-10-29.1
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.