Skip to main content
Request access

CRAWLER DIRECTORY

Every crawler,
and how to check it.

72 bots and agents from 36 operators, from AI training crawlers to search engines and SEO tools. Each profile gives the exact robots.txt rule, what blocking it costs and how to verify that a visit is real.

Search all bots
See the operators
  • From the operators’ own documentation
  • A verification method for every bot
  • The registry behind Crawl Analytics
Categories72 bots
  • 29 botsAI botsCrawlers and fetchers behind AI products: model training, AI search and assistants that fetch a page for a user
  • 16 botsSearch enginesCrawlers that index your pages for search results
  • 10 botsSEO toolsCrawlers that SEO platforms run to build backlink, keyword and audit data
  • 3 botsSocial previewsFetchers that build the preview card when someone shares your page in a social or chat app
  • 14 botsOther agentsAdvertising checks, monitors, security scanners and scripts that identify themselves loosely or not at all

72

bots and agents

36

operators

44

can be verified

5

categories

BROWSE BY CATEGORY

Start with what the crawler does.

The purpose, not the company, decides whether to allow it.

29 BOTS

AI bots

Crawlers and fetchers behind AI products: model training, AI search and assistants that fetch a page for a user.

Browse ai bots

16 BOTS

Search engines

Crawlers that index your pages for search results. Blocking one removes your pages from its results.

Browse search engines

10 BOTS

SEO tools

Crawlers that SEO platforms run to build backlink, keyword and audit data. Blocking them does not change your rankings.

Browse seo tools

3 BOTS

Social previews

Fetchers that build the preview card when someone shares your page in a social or chat app.

Browse social previews

14 BOTS

Other agents

Advertising checks, monitors, security scanners and scripts that identify themselves loosely or not at all.

Browse other agents

ALL BOTS

Search every crawler.

Filter by purpose, verification or operator, or search for the name in your logs.

72 bots

GoogleVerifiable

AdsBot-Google

AdsBot-Google AdsBot-Google-Mobile

  • Advertising

Follows robots.txt

AhrefsVerifiable

AhrefsBot

AhrefsBot AhrefsSiteAudit

  • SEO tool

Follows robots.txt

Allen Institute for AIUnverified

AI2Bot

AI2Bot

  • AI training

Not documented

AmazonVerifiable

Amazonbot

Amazonbot

  • AI search retrieval
  • Data collection

Follows robots.txt

AmazonUnverified

Amzn-SearchBot

Amzn-SearchBot

  • AI search retrieval

Follows robots.txt

AmazonUnverified

Amzn-User

Amzn-User

  • User-directed fetch

User-triggered

AppleVerifiable

Applebot

Applebot

  • Search indexing

Follows robots.txt

AppleControl token

Applebot-Extended

Applebot-Extended

  • AI training

Follows robots.txt

BaiduVerifiable

Baiduspider

Baiduspider

  • Search indexing

Follows robots.txt

BabbarVerifiable

Barkrowler

barkrowler

  • SEO tool

Follows robots.txt

MicrosoftVerifiable

Bingbot

Bingbot msnbot AdIdxBot

  • Search indexing
  • AI search retrieval

Follows robots.txt

ByteDanceUnverified

Bytespider

Bytespider

  • AI training
  • Data collection

Not documented

Common CrawlVerifiable

CCBot

CCBot

  • AI training
  • Data collection

Follows robots.txt

OpenAIVerifiable

ChatGPT agent

No robots.txt token

  • Transactional agent
  • User-directed fetch

User-triggered

OpenAIVerifiable

ChatGPT-User

ChatGPT-User

  • User-directed fetch

User-triggered

AnthropicVerifiable

Claude-SearchBot

Claude-SearchBot

  • AI search retrieval

Follows robots.txt

AnthropicVerifiable

Claude-User

Claude-User Claude-Web

  • User-directed fetch

Follows robots.txt

AnthropicVerifiable

ClaudeBot

ClaudeBot anthropic-ai

  • AI training

Follows robots.txt

Cốc CốcVerifiable

coccocbot

coccocbot coccocbot-web coccocbot-image

  • Search indexing

Follows robots.txt

CohereUnverified

cohere-ai

cohere-ai cohere-training-data-crawler

  • User-directed fetch
  • AI training

Not documented

DataForSEOUnverified

DataForSeoBot

DataForSeoBot

  • SEO tool

Follows robots.txt

DiffbotUnverified

Diffbot

Diffbot

  • Data collection

Not documented

MozUnverified

DotBot

DotBot rogerbot

  • SEO tool

Follows robots.txt

DuckDuckGoVerifiable

DuckAssistBot

DuckAssistBot

  • AI search retrieval
  • User-directed fetch

Follows robots.txt

DuckDuckGoVerifiable

DuckDuckBot

DuckDuckBot

  • Search indexing

Follows robots.txt

GoogleVerifiable

Google-Agent

No robots.txt token

  • Transactional agent
  • User-directed fetch

User-triggered

GoogleControl token

Google-Extended

Google-Extended

  • AI training

Follows robots.txt

GoogleVerifiable

Google-InspectionTool

Google-InspectionTool Googlebot

  • Search indexing

Follows robots.txt

GoogleVerifiable

Google-Safety

No robots.txt token

  • Security scanner

Ignores robots.txt

GoogleVerifiable

Googlebot

Googlebot

  • Search indexing

Follows robots.txt

GoogleVerifiable

Googlebot-Image

Googlebot-Image Googlebot

  • Search indexing

Follows robots.txt

GoogleControl token

Googlebot-News

Googlebot-News

  • Search indexing

Follows robots.txt

GoogleVerifiable

Googlebot-Video

Googlebot-Video Googlebot

  • Search indexing

Follows robots.txt

GoogleVerifiable

GoogleOther

GoogleOther

  • Data collection

Follows robots.txt

GoogleVerifiable

GoogleOther-Image

GoogleOther-Image GoogleOther

  • Data collection

Follows robots.txt

GoogleVerifiable

GoogleOther-Video

GoogleOther-Video GoogleOther

  • Data collection

Follows robots.txt

OpenAIVerifiable

GPTBot

GPTBot

  • AI training

Follows robots.txt

HiveUnverified

ImagesiftBot

ImagesiftBot

  • Data collection

Follows robots.txt

KagiUnverified

Kagibot

Kagibot

  • Search indexing

Follows robots.txt

MetaUnverified

Meta-ExternalAds

meta-externalads

  • Advertising

Follows robots.txt

MetaVerifiable

Meta-ExternalAgent

meta-externalagent FacebookBot

  • AI training
  • Data collection

Follows robots.txt

MetaUnverified

Meta-WebIndexer

meta-webindexer

  • AI search retrieval

Follows robots.txt

Mistral AIUnverified

MistralAI-User

MistralAI-User

  • User-directed fetch

User-triggered

MajesticUnverified

MJ12bot

MJ12bot

  • SEO tool

Follows robots.txt

NdovuSEOLabsVerifiable

NdovuBot

NdovuBot OpenSEO-Audit

  • NdovuSEOLabs tool
  • SEO tool

Follows robots.txt

OpenAIVerifiable

OAI-AdsBot

OAI-AdsBot

  • Advertising

Not documented

OpenAIVerifiable

OAI-SearchBot

OAI-SearchBot

  • AI search retrieval

Follows robots.txt

PerplexityVerifiable

Perplexity-User

Perplexity-User

  • User-directed fetch

User-triggered

PerplexityVerifiable

PerplexityBot

PerplexityBot

  • AI search retrieval

Follows robots.txt

HuaweiVerifiable

PetalBot

PetalBot

  • Search indexing

Follows robots.txt

PinterestVerifiable

Pinterestbot

Pinterestbot

  • Social preview

Follows robots.txt

VariousUnverified

Security scanners

No robots.txt token

  • Security scanner

Ignores robots.txt

SISTRIXUnverified

SeekportBot

SeekportBot

  • Search indexing

Follows robots.txt

SemrushVerifiable

SemrushBot

SemrushBot SiteAuditBot

  • SEO tool

Follows robots.txt

SeobilityVerifiable

SeobilityBot

Seobility

  • SEO tool

Follows robots.txt

SerpstatUnverified

serpstatbot

serpstatbot

  • SEO tool

Follows robots.txt

SeznamVerifiable

SeznamBot

SeznamBot

  • Search indexing

Follows robots.txt

GoogleVerifiable

Storebot-Google

Storebot-Google

  • Search indexing

Follows robots.txt

VariousUnverified

Uptime monitors

No robots.txt token

  • Monitoring

Ignores robots.txt

Webz.ioUnverified

Webzio

Webzio Omgilibot

  • Data collection

Follows robots.txt

Webz.ioControl token

Webzio-Extended

Webzio-Extended

  • AI training

Follows robots.txt

YandexVerifiable

YandexBot

Yandex YandexBot

  • Search indexing

Follows robots.txt

You.comUnverified

YouBot

YouBot

  • AI search retrieval

Follows robots.txt

HOW TO READ A PROFILE

Four things every profile tells you.

Enough to decide, and to check what your logs claim.

TOKEN

The robots.txt token

The exact name to put in a User-agent line, with copy-ready rules to block or allow it.

VERIFY

How to verify a visit

Published IP ranges, reverse DNS or signed requests. A user agent alone is only a claim.

ROBOTS

Whether it obeys robots.txt

Some bots fetch only when a person asks and may not follow robots.txt; a few ignore it entirely.

IMPACT

What blocking it costs

Training crawlers can go without touching rankings; search crawlers cannot.

FAQ

About the crawler directory.

Where the data comes from, and what it can tell you.

Where does this data come from?

From the registry NdovuSEOLabs uses to classify crawler traffic in Crawl Analytics. Each entry is built from the operator’s own documentation, and each profile shows when that was last checked. Registry version 2026-10-29.1.

What does “can be verified” mean?

The operator publishes a way to check that a request really comes from its bot: IP ranges, reverse DNS or signed requests. A user agent alone is only a claim, and anyone can copy one.

Does blocking AI training crawlers hurt my SEO?

Blocking a training crawler such as GPTBot or ClaudeBot does not change your search rankings. Blocking a search crawler such as Googlebot or Bingbot removes your pages from those results, and blocking AI search crawlers can keep you out of AI answers.

Why is a bot I see in my logs missing?

The directory lists the bots NdovuSEOLabs recognises, and adds one when its operator documents it, so every entry can be checked.

How do I see which bots visit my site?

Install the WordPress plugin or connect your server or CDN logs. Crawl Analytics then counts each bot’s requests and separates verified visits from impostors.

See which of these crawlers visit your site.

Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.

Explore Crawl Analytics