AI BOTS · DIFFBOT
Diffbot
Crawlers and fetchers behind AI products: model training, AI search and assistants that fetch a page for a user.
- robots.txt behaviour not documented
- Cannot be verified
- No public documentation
- UnverifiedIdentityNo published check
- Unknownrobots.txtrobots.txt behaviour not documented
- PurposeData collectionFor an AI product
- TokenDiffbotUse it in a User-agent line
robots.txt
The operator does not say whether this bot follows robots.txt.
Block Diffbot
User-agent: Diffbot
Disallow: /Allow Diffbot
User-agent: Diffbot
Allow: /Quick facts
If you block it
Keeps your pages out of what this operator collects.
Check a visit is real
- No published way to check it. A request using this name is only a claim.
Where this comes from
The same registry NdovuSEOLabs uses to classify crawler traffic in your server and CDN logs.
Registry version 2026-10-29.1
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.