Skip to main content
Request access

CRAWLER OPERATOR

Meta crawlers.

Meta runs 5 bots in this directory: Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds, facebookexternalhit. Each can be allowed or blocked on its own.

See its bots
All operators
  • 5 bots
  • 3 can be verified
  • 3 for AI products
Meta5 bots
  • VerifiableMeta-ExternalAgentAI training, Data collection
  • VerifiableMeta-ExternalFetcherUser-directed fetch
  • UnverifiedMeta-WebIndexerAI search retrieval
  • UnverifiedMeta-ExternalAdsAdvertising
  • VerifiablefacebookexternalhitSocial preview

META

Every Meta bot.

Open a bot for its robots.txt rule, verification and documentation.

5 bots

MetaUnverified

Meta-ExternalAds

meta-externalads

  • Advertising

Follows robots.txt

MetaVerifiable

Meta-ExternalAgent

meta-externalagent FacebookBot

  • AI training
  • Data collection

Follows robots.txt

MetaUnverified

Meta-WebIndexer

meta-webindexer

  • AI search retrieval

Follows robots.txt

See which of these crawlers visit your site.

Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.

Explore Crawl Analytics