Skip to main content
Request access

CRAWLER DIRECTORY · AI BOTS

AI bots.

Crawlers and fetchers behind AI products: model training, AI search and assistants that fetch a page for a user.

Browse the list
All categories
  • 29 bots
  • 17 can be verified
  • 17 follow robots.txt
AI botsA–Z
  • UnverifiedAI2BotAllen Institute for AI · AI training
  • VerifiableAmazonbotAmazon · AI search retrieval, Data collection
  • UnverifiedAmzn-SearchBotAmazon · AI search retrieval
  • UnverifiedAmzn-UserAmazon · User-directed fetch

29

bots

17

can be verified

17

follow robots.txt

16

operators

AI BOTS

All 29 bots.

Search by name or token, or filter by purpose, verification and operator.

29 bots

Allen Institute for AIUnverified

AI2Bot

AI2Bot

  • AI training

Not documented

AmazonVerifiable

Amazonbot

Amazonbot

  • AI search retrieval
  • Data collection

Follows robots.txt

AmazonUnverified

Amzn-SearchBot

Amzn-SearchBot

  • AI search retrieval

Follows robots.txt

AmazonUnverified

Amzn-User

Amzn-User

  • User-directed fetch

User-triggered

AppleControl token

Applebot-Extended

Applebot-Extended

  • AI training

Follows robots.txt

ByteDanceUnverified

Bytespider

Bytespider

  • AI training
  • Data collection

Not documented

Common CrawlVerifiable

CCBot

CCBot

  • AI training
  • Data collection

Follows robots.txt

OpenAIVerifiable

ChatGPT agent

No robots.txt token

  • Transactional agent
  • User-directed fetch

User-triggered

OpenAIVerifiable

ChatGPT-User

ChatGPT-User

  • User-directed fetch

User-triggered

AnthropicVerifiable

Claude-SearchBot

Claude-SearchBot

  • AI search retrieval

Follows robots.txt

AnthropicVerifiable

Claude-User

Claude-User Claude-Web

  • User-directed fetch

Follows robots.txt

AnthropicVerifiable

ClaudeBot

ClaudeBot anthropic-ai

  • AI training

Follows robots.txt

CohereUnverified

cohere-ai

cohere-ai cohere-training-data-crawler

  • User-directed fetch
  • AI training

Not documented

DiffbotUnverified

Diffbot

Diffbot

  • Data collection

Not documented

DuckDuckGoVerifiable

DuckAssistBot

DuckAssistBot

  • AI search retrieval
  • User-directed fetch

Follows robots.txt

GoogleVerifiable

Google-Agent

No robots.txt token

  • Transactional agent
  • User-directed fetch

User-triggered

GoogleControl token

Google-Extended

Google-Extended

  • AI training

Follows robots.txt

OpenAIVerifiable

GPTBot

GPTBot

  • AI training

Follows robots.txt

MetaVerifiable

Meta-ExternalAgent

meta-externalagent FacebookBot

  • AI training
  • Data collection

Follows robots.txt

MetaUnverified

Meta-WebIndexer

meta-webindexer

  • AI search retrieval

Follows robots.txt

Mistral AIUnverified

MistralAI-User

MistralAI-User

  • User-directed fetch

User-triggered

OpenAIVerifiable

OAI-AdsBot

OAI-AdsBot

  • Advertising

Not documented

OpenAIVerifiable

OAI-SearchBot

OAI-SearchBot

  • AI search retrieval

Follows robots.txt

PerplexityVerifiable

Perplexity-User

Perplexity-User

  • User-directed fetch

User-triggered

PerplexityVerifiable

PerplexityBot

PerplexityBot

  • AI search retrieval

Follows robots.txt

Webz.ioControl token

Webzio-Extended

Webzio-Extended

  • AI training

Follows robots.txt

You.comUnverified

YouBot

YouBot

  • AI search retrieval

Follows robots.txt

FAQ

AI bots: common questions.

Before you change your robots.txt.

Should I block ai bots?

It depends on the purpose. Training crawlers can be blocked without changing your rankings. AI search crawlers and assistants decide whether your pages can appear in AI answers, so blocking them can cost visibility.

How do I block one?

Open its profile and copy the robots.txt rule. Bots that ignore robots.txt, or fetch only when a person asks, need a server, firewall or CDN rule instead.

See which of these crawlers visit your site.

Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.

Explore Crawl Analytics