TECHNICAL SEO
Robots.txt Validator
A robots.txt mistake can hide a whole site from crawlers or leave the rules you wrote unapplied. Paste the file to check every line against the robots.txt standard and Google’s documented behaviour.
- Runs in your browser
- Nothing you paste is uploaded
- Free, no sign-up
- CheckLines without a colon
- CheckAllow or Disallow before any User-agent
- CheckRelative or invalid Sitemap URLs
- CheckNoindex lines Google no longer supports
Free
no sign-up, no usage limit
0 uploads
what you paste stays on your device
7 checks
what the tool looks at
Offline
works once the page has loaded
Loading the tool…
HOW IT WORKS
How to use the Robots.txt Validator
The contents of a robots.txt file.
Open yoursite.co.ke/robots.txt and copy its contents.
Paste it into the validator.
Fix errors first, then review warnings and notes.
WHAT IT CHECKS
What it looks at, and how to read it.
Findings by line and severity, the user-agent groups with their rule counts, sitemaps and file size.
What it checks
- Lines without a colon
- Allow or Disallow before any User-agent
- Relative or invalid Sitemap URLs
- Noindex lines Google no longer supports
- Crawl-delay values and which engines read them
- Unknown directives and file size against Google’s 500 KiB limit
- Whether a group for all crawlers exists
How to read the results
- Errors are lines crawlers ignore or misread.
- Warnings are lines that may not do what you expect.
- Notes explain engine differences, such as Crawl-delay.
The concepts behind it, in the SEO glossary: Robots.txt.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Using robots.txt to keep pages out of search results; use noindex instead.
- Blocking CSS and JavaScript that pages need to render.
- Writing a relative Sitemap URL.
A worked example
A WooCommerce store’s file has "noindex: /old-promo/" and "Sitemap: /sitemap.xml". The validator flags both: the noindex line is ignored and the sitemap must be a full URL.
Limitations
- Validation follows RFC 9309 and Google’s documentation; other crawlers may read edge cases differently.
- Fetching a live robots.txt from a URL arrives with the public tools API; for now, paste the file.
FAQ
Questions about the Robots.txt Validator
Straight answers about what it does and where its limits are.
Does blocking a page in robots.txt remove it from Google?
No. It stops crawling. A blocked page can still be indexed if other pages link to it. Use a noindex directive on a crawlable page to remove it.
See which crawlers actually visit in Request Intelligence.
Free tools check one thing at a time. The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and keeps checking every page.