Skip to main content
Request access

CRAWLER DIRECTORY · OPERATORS

The companies
behind the crawlers.

36 operators and the bots each one runs. Large operators run separate bots for search, AI training and assistants, and each can be allowed or blocked on its own.

Browse operators
Search all bots
  • 36 operators
  • 72 bots
  • Linked to official documentation
Most botsBy operator
  • 16 botsGoogleGooglebot-Image, Googlebot-Video, Google-InspectionTool
  • 5 botsMetaMeta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer
  • 5 botsOpenAIGPTBot, OAI-SearchBot, ChatGPT-User
  • 3 botsAmazonAmazonbot, Amzn-SearchBot, Amzn-User

ALL OPERATORS

Look up a company.

Open an operator to see every bot it runs and what each one does.

16 bots3 AI

Google

Googlebot-Image, Googlebot-Video, Google-InspectionTool, Storebot-Google, GoogleOther-Image, GoogleOther-Video, GoogleOther, Google-CloudVertexBot, AdsBot-Google, Mediapartners-Google, Google-Agent, Google-Safety, Google user-triggered fetchers, Googlebot, Googlebot-News, Google-Extended

14 of 16 can be verified

5 bots3 AI

Meta

Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds, facebookexternalhit

3 of 5 can be verified

5 bots5 AI

OpenAI

GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ChatGPT agent

5 of 5 can be verified

3 bots3 AI

Amazon

Amazonbot, Amzn-SearchBot, Amzn-User

1 of 3 can be verified

3 bots3 AI

Anthropic

ClaudeBot, Claude-SearchBot, Claude-User

3 of 3 can be verified

2 bots1 AI

Apple

Applebot, Applebot-Extended

1 of 2 can be verified

2 bots1 AI

DuckDuckGo

DuckDuckBot, DuckAssistBot

2 of 2 can be verified

2 bots2 AI

Perplexity

PerplexityBot, Perplexity-User

2 of 2 can be verified

2 bots1 AI

Webz.io

Webzio, Webzio-Extended

0 of 2 can be verified

1 bot

Ahrefs

AhrefsBot

1 of 1 can be verified

1 bot

Babbar

Barkrowler

1 of 1 can be verified

1 bot

Baidu

Baiduspider

1 of 1 can be verified

1 bot1 AI

ByteDance

Bytespider

0 of 1 can be verified

1 bot1 AI

Cohere

cohere-ai

0 of 1 can be verified

1 bot

DataForSEO

DataForSeoBot

0 of 1 can be verified

1 bot1 AI

Diffbot

Diffbot

0 of 1 can be verified

1 bot

Hive

ImagesiftBot

0 of 1 can be verified

1 bot

Huawei

PetalBot

1 of 1 can be verified

1 bot

Kagi

Kagibot

0 of 1 can be verified

1 bot

Majestic

MJ12bot

0 of 1 can be verified

1 bot

Microsoft

Bingbot

1 of 1 can be verified

1 bot1 AI

Mistral AI

MistralAI-User

0 of 1 can be verified

1 bot

Moz

DotBot

0 of 1 can be verified

1 bot

Pinterest

Pinterestbot

1 of 1 can be verified

1 bot

Screaming Frog

Screaming Frog SEO Spider

0 of 1 can be verified

1 bot

Semrush

SemrushBot

1 of 1 can be verified

1 bot

Seobility

SeobilityBot

1 of 1 can be verified

1 bot

Serpstat

serpstatbot

0 of 1 can be verified

1 bot

Seznam

SeznamBot

1 of 1 can be verified

1 bot

SISTRIX

SeekportBot

0 of 1 can be verified

1 bot

Yandex

YandexBot

1 of 1 can be verified

1 bot1 AI

You.com

YouBot

0 of 1 can be verified

See which of these crawlers visit your site.

Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.

Explore Crawl Analytics