CRAWLER DIRECTORY · OTHER AGENTS
Other agents.
Advertising checks, monitors, security scanners and scripts that identify themselves loosely or not at all.
- 14 bots
- 7 can be verified
- 8 follow robots.txt
- VerifiableAdsBot-GoogleGoogle · Advertising
- VerifiableGoogle user-triggered fetchersGoogle · Feed reader, User-directed fetch
- VerifiableGoogle-SafetyGoogle · Security scanner
- VerifiableGoogleOtherGoogle · Data collection
14
bots
7
can be verified
8
follow robots.txt
6
operators
OTHER AGENTS
All 14 bots.
Search by name or token, or filter by purpose, verification and operator.
14 bots
AdsBot-Google
AdsBot-Google AdsBot-Google-Mobile
Follows robots.txt
Google user-triggered fetchers
No robots.txt token
User-triggered
Google-Safety
No robots.txt token
Ignores robots.txt
GoogleOther
GoogleOther
Follows robots.txt
GoogleOther-Image
GoogleOther-Image GoogleOther
Follows robots.txt
GoogleOther-Video
GoogleOther-Video GoogleOther
Follows robots.txt
ImagesiftBot
ImagesiftBot
Follows robots.txt
Mediapartners-Google
Mediapartners-Google
Follows robots.txt
Meta-ExternalAds
meta-externalads
Follows robots.txt
Other self-declared bots
No robots.txt token
Not documented
Scripts and headless browsers
No robots.txt token
Not documented
Security scanners
No robots.txt token
Ignores robots.txt
Uptime monitors
No robots.txt token
Ignores robots.txt
Webzio
Webzio Omgilibot
Follows robots.txt
No bots match these filters.
FAQ
Other agents: common questions.
Before you change your robots.txt.
Should I block other agents?
Advertising checks are worth allowing if you run ads. Scanners and anonymous scripts ignore robots.txt, so manage them at the server, firewall or CDN.
How do I block one?
Open its profile and copy the robots.txt rule. Bots that ignore robots.txt, or fetch only when a person asks, need a server, firewall or CDN rule instead.
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.