TECHNICAL SEO
Robots.txt Generator
A good robots.txt is short: it keeps crawlers out of admin, cart and internal search pages and points them to the sitemap. Most mistakes come from blocking too much, such as CSS files or whole sections. This generator starts from a sensible preset and checks the result.
- Runs in your browser
- Nothing you paste is uploaded
- Free, no sign-up
- CheckPlatform presets for WordPress and WooCommerce
- CheckInternal search and cart pages
- CheckCustom paths, written as valid rules
- CheckSitemap lines as full URLs
Free
no sign-up, no usage limit
0 uploads
what you paste stays on your device
5 checks
what the tool looks at
Offline
works once the page has loaded
Loading the tool…
HOW IT WORKS
How to use the Robots.txt Generator
A platform, options and the paths to block.
Choose your platform.
Add any paths that should never be crawled, and your sitemap URLs.
Download the file, upload it to your site root and test important URLs with the Robots.txt URL Tester.
WHAT IT CHECKS
What it looks at, and how to read it.
A robots.txt file to copy or download, and the validator’s findings.
What it checks
- Platform presets for WordPress and WooCommerce
- Internal search and cart pages
- Custom paths, written as valid rules
- Sitemap lines as full URLs
- Syntax checked by the validator
How to read the results
- Blocking a URL in robots.txt hides its noindex from crawlers, so it can still be indexed from links.
The concepts behind it, in the SEO glossary: Robots.txt.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Leaving Disallow: / from staging in place after launch.
- Blocking /wp-content/, which holds the CSS and images Google needs.
A worked example
A WooCommerce store blocks cart, checkout, account, add-to-cart and sort URLs and lists its sitemap index.
Limitations
- robots.txt stops crawling, not indexing; it is not a way to remove pages from search.
- For AI crawlers, use the AI Robots.txt Policy Generator.
FAQ
Questions about the Robots.txt Generator
Straight answers about what it does and where its limits are.
Where does robots.txt go?
At the root of each host: https://yourstore.co.ke/robots.txt. A subdomain needs its own file.
Need this for a whole website, over time?
The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and audits every page on a schedule.