SEO TOOLS · SCREAMING FROG
Screaming Frog SEO Spider
A desktop crawler anyone can run, often with a changed user agent.
- Follows robots.txt
- Cannot be verified
- No public documentation
- UnverifiedIdentityNo published check
- Obeysrobots.txtFollows robots.txt
- PurposeSEO toolNot an AI product
- TokenScreaming Frog SEO SpiderUse it in a User-agent line
robots.txt
The operator documents that this bot follows robots.txt rules.
Block Screaming Frog SEO Spider
User-agent: Screaming Frog SEO Spider
Disallow: /Allow Screaming Frog SEO Spider
User-agent: Screaming Frog SEO Spider
Allow: /Quick facts
- robots.txt token
Screaming Frog SEO Spider- User agent contains
- Screaming Frog SEO Spider
- Operator
- Screaming Frog
- Category
- SEO tools
- Purpose
- SEO tool
- robots.txt
- Follows robots.txt
- Verification
- Cannot be verified
If you block it
This SEO tool stops collecting data about your site. Your rankings are not affected.
Check a visit is real
- No published way to check it. A request using this name is only a claim.
Where this comes from
The same registry NdovuSEOLabs uses to classify crawler traffic in your server and CDN logs.
Registry version 2026-10-29.1
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.