Skip to main content
Request access

CRAWLING, RENDERING & INDEXING

Robots.txt: what it does, and what it cannot do

The file at /robots.txt that tells crawlers which URLs not to fetch; it controls crawling, not indexing.

Check it on your site
All glossary terms
  • Beginner level
  • 2 primary sources
  • Reviewed 29 October 2026
At a glanceSEO glossary
  • TopicCrawling, rendering & indexingAlso in Technical SEO, AI search & generative search
  • LevelBeginnerNo background needed
  • Sources2 primary sourcesGoogle Search Central, IETF
  • Reviewed29 October 2026Re-checked every 12 months
On this page
  1. Quick definition
  2. Why it matters
  3. How it works
  4. How to check it
  5. Common mistakes
  6. Related concepts
  7. Check it on your site
  8. Keep learning
  9. Sources and review

Quick definition

Robots.txt is a plain-text file at the root of a host that tells crawlers which URLs they may request. It controls crawling, not indexing: a URL blocked in robots.txt can still be indexed, without its content, if other pages link to it.

Why it matters

Robots.txt is the cheapest way to keep crawlers out of URL spaces that waste their time, such as internal search, carts or endless filter combinations. It is also how site owners state which AI crawlers may fetch their content.

A single wrong line can block a whole site. Disallow: / under User-agent: * stops compliant crawlers from fetching anything.

How it works

The file lives at /robots.txt on each protocol and host combination: https://www.example.com/robots.txt does not cover https://shop.example.com/. Rules are grouped by User-agent. A crawler uses the most specific group that names it and ignores the others; if none names it, it uses the * group.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml
Everyone is kept out of the cart and internal search (except a help page); GPTBot is kept out of the whole site.

Within a group, Google applies the most specific (longest) matching rule; when an Allow and a Disallow match equally, Allow wins. The protocol was standardised as RFC 9309 in 2022. Google reads up to 500 KiB of the file, and compliance is voluntary: well-behaved crawlers obey it, scrapers may not.

How to check it

  1. Open /robots.txt on every host and protocol you serve. Confirm it returns 200 (or 404 if you intend no rules) and not a 5xx error, which can make Google pause crawling.
  2. Test the important URLs against the rules for each crawler that matters: Googlebot, Bingbot and the AI crawlers you have a policy for.
  3. Search Console‘s robots.txt report shows the versions Google fetched and any parse errors.
  4. Check that CSS and JavaScript needed to render pages are not disallowed.

Common mistakes and misconceptions

  • Myth Robots.txt keeps a page out of search results.

    In fact It stops crawling. To keep a page out of the index, allow crawling and use noindex, or protect the page with a login.

  • Myth Noindex: lines in robots.txt work.

    In fact Google stopped supporting noindex rules in robots.txt in 2019.

  • Myth Robots.txt is a security control.

    In fact The file is public and voluntary. Listing private paths in it advertises them.

Check it on your site

Free tools, no sign-up

In the NdovuSEOLabs platform

Run the same checks on every page of your site, with your own data.

Keep learning

Sources and review

Written and maintained by Ndovu SEO Labs. First published 29 October 2026. Facts last checked against the sources above on 29 October 2026. How we write and review the glossary.