CRAWLER DIRECTORY · SEARCH ENGINES
Search engines.
Crawlers that index your pages for search results. Blocking one removes your pages from its results.
- 16 bots
- 13 can be verified
- 16 follow robots.txt
- VerifiableApplebotApple · Search indexing
- VerifiableBaiduspiderBaidu · Search indexing
- VerifiableBingbotMicrosoft · Search indexing, AI search retrieval
- VerifiablecoccocbotCốc Cốc · Search indexing
16
bots
13
can be verified
16
follow robots.txt
11
operators
SEARCH ENGINES
All 16 bots.
Search by name or token, or filter by purpose, verification and operator.
16 bots
Applebot
Applebot
Follows robots.txt
Baiduspider
Baiduspider
Follows robots.txt
Bingbot
Bingbot msnbot AdIdxBot
Follows robots.txt
coccocbot
coccocbot coccocbot-web coccocbot-image
Follows robots.txt
DuckDuckBot
DuckDuckBot
Follows robots.txt
Google-InspectionTool
Google-InspectionTool Googlebot
Follows robots.txt
Googlebot
Googlebot
Follows robots.txt
Googlebot-Image
Googlebot-Image Googlebot
Follows robots.txt
Googlebot-News
Googlebot-News
Follows robots.txt
Googlebot-Video
Googlebot-Video Googlebot
Follows robots.txt
Kagibot
Kagibot
Follows robots.txt
PetalBot
PetalBot
Follows robots.txt
SeekportBot
SeekportBot
Follows robots.txt
SeznamBot
SeznamBot
Follows robots.txt
Storebot-Google
Storebot-Google
Follows robots.txt
YandexBot
Yandex YandexBot
Follows robots.txt
No bots match these filters.
FAQ
Search engines: common questions.
Before you change your robots.txt.
Should I block search engines?
Rarely. Blocking a search crawler removes your pages from that engine’s results. Block one only for a market you do not serve, or a crawler that misbehaves.
How do I block one?
Open its profile and copy the robots.txt rule. Bots that ignore robots.txt, or fetch only when a person asks, need a server, firewall or CDN rule instead.
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.