CRAWLER DIRECTORY · MONITORING BOTS
Monitoring bots.
Uptime checkers, redirect services, compliance scanners and brand monitors that load the pages someone asked them to watch.
MONITORING BOTS
All 16 named bots.
Search by name or token, or filter by purpose, verification and operator.
16 bots
AwarioBot
AwarioBot AwarioSmartBot AwarioRssBot
Follows robots.txt
Better Stack
No robots.txt token
User-triggered
Checkly
No robots.txt token
User-triggered
Datadog Synthetics
No robots.txt token
User-triggered
FreshpingBot
No robots.txt token
User-triggered
HetrixTools
No robots.txt token
User-triggered
Pingdom
No robots.txt token
User-triggered
QATechBot
QATechBot
Follows robots.txt
redirect.pizza
No robots.txt token
Not documented
ScooterZA
ScooterZA
Follows robots.txt
Site24x7
No robots.txt token
User-triggered
StatusCake
No robots.txt token
User-triggered
TermlyBot
TermlyBot
Follows robots.txt
UptimeRobot
No robots.txt token
User-triggered
WARDBot
WARDBot
Follows robots.txt
Yandex page checkers
No robots.txt token
Ignores robots.txt
No bots match these filters.
Should I block monitoring bots?
Usually not. Most check pages that you, or a service you use, asked them to watch, and they do not read robots.txt. To stop one, remove the check in that service or block it at the server.
How do I block one?
Open its profile and copy the robots.txt rule. Bots that ignore robots.txt, or fetch only when a person asks, need a server, firewall or CDN rule instead.
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.