CRAWLER DIRECTORY · SEO TOOLS
SEO tools.
Crawlers that SEO platforms run to build backlink, keyword and audit data. Blocking them does not change your rankings.
- 10 bots
- 5 can be verified
- 10 follow robots.txt
- VerifiableAhrefsBotAhrefs · SEO tool
- VerifiableBarkrowlerBabbar · SEO tool
- UnverifiedDataForSeoBotDataForSEO · SEO tool
- UnverifiedDotBotMoz · SEO tool
10
bots
5
can be verified
10
follow robots.txt
10
operators
SEO TOOLS
All 10 bots.
Search by name or token, or filter by purpose, verification and operator.
10 bots
AhrefsBot
AhrefsBot AhrefsSiteAudit
Follows robots.txt
Barkrowler
barkrowler
Follows robots.txt
DataForSeoBot
DataForSeoBot
Follows robots.txt
DotBot
DotBot rogerbot
Follows robots.txt
MJ12bot
MJ12bot
Follows robots.txt
NdovuBot
NdovuBot OpenSEO-Audit
Follows robots.txt
Screaming Frog SEO Spider
Screaming Frog SEO Spider
Follows robots.txt
SemrushBot
SemrushBot SiteAuditBot
Follows robots.txt
SeobilityBot
Seobility
Follows robots.txt
serpstatbot
serpstatbot
Follows robots.txt
No bots match these filters.
FAQ
SEO tools: common questions.
Before you change your robots.txt.
Should I block seo tools?
You can block them without changing your rankings. They stop collecting backlink, keyword and audit data about your site, which also hides it from anyone using those tools. Keep your own audit crawler allowed.
How do I block one?
Open its profile and copy the robots.txt rule. Bots that ignore robots.txt, or fetch only when a person asks, need a server, firewall or CDN rule instead.
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.