TECHNICAL SEO
Robots.txt URL Tester
When a page does not get crawled, or an AI crawler still fetches content you meant to block, you need to know which rule applies. Test any list of URLs against a robots.txt and see the deciding line for each crawler.
- Runs in your browser
- Nothing you paste is uploaded
- Free, no sign-up
- CheckUser-agent group selection, including fallback to *
- CheckLongest-match precedence between Allow and Disallow
- CheckWildcards (*) and end-of-URL anchors ($)
- CheckQuery strings in paths
Free
no sign-up, no usage limit
0 uploads
what you paste stays on your device
4 checks
what the tool looks at
Offline
works once the page has loaded
Loading the tool…
HOW IT WORKS
How to use the Robots.txt URL Tester
A robots.txt file, the site address, a crawler and the paths to test.
Paste the robots.txt and enter your site’s address.
Choose a crawler or type any user-agent token.
Add the paths to test, one per line.
WHAT IT CHECKS
What it looks at, and how to read it.
Allowed or blocked for each URL, with the line number and rule that decided it.
What it checks
- User-agent group selection, including fallback to *
- Longest-match precedence between Allow and Disallow
- Wildcards (*) and end-of-URL anchors ($)
- Query strings in paths
How to read the results
- Blocked means the crawler should not fetch the URL.
- "No rule matches" means the URL is allowed by default.
- A crawler with its own group ignores the * group entirely.
The concepts behind it, in the SEO glossary: Web crawler, Robots.txt.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Expecting a GPTBot group to inherit rules from User-agent: *.
- Forgetting that paths are case-sensitive.
A worked example
With "Disallow: /cart" and "Allow: /cart/share", the tester shows /cart/share/123 allowed for Googlebot by the longer Allow rule, while GPTBot is blocked everywhere by its own group.
Limitations
- Matching follows Google’s rules: the most specific user-agent group, the longest matching path, Allow winning a tie. Some crawlers implement edge cases differently.
FAQ
Questions about the Robots.txt URL Tester
Straight answers about what it does and where its limits are.
Why is GPTBot blocked when the * group allows the page?
A crawler that has its own User-agent group follows only that group. If the GPTBot group says Disallow: /, nothing from the * group applies to it.
Compare robots rules with real crawler visits.
Free tools check one thing at a time. The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and keeps checking every page.