Robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may or may not request. It controls crawling, not indexing. That distinction matters because a page blocked in robots.txt can still appear in search results if other pages link to it, just without a description. If your goal is to keep a page out of the index, robots.txt is the wrong tool, and reaching for it is one of the more common ways I see otherwise careful teams shoot themselves in the foot.

What robots.txt is actually for

I use robots.txt to manage crawl efficiency, not to hide content. On a large site, steering crawlers away from low-value paths leaves more crawl budget for the pages that actually need to be discovered and refreshed often.

  • Blocking faceted navigation, internal search results, and the near-infinite filter combinations that crawlers can otherwise get lost in.
  • Keeping crawlers out of admin areas, carts, checkout steps, and staging paths that serve no purpose in search.
  • Pointing to your XML sitemap with a Sitemap: directive, which is a small but genuinely useful courtesy.

The crawl budget argument becomes real at scale. A mid-size e-commerce site with 50,000 product pages can easily generate millions of crawlable URLs through faceted navigation: every combination of color, size, brand, and price range gets its own URL. Googlebot will spend its entire crawl allocation on those parameter pages, refreshing the same filtered views over and over, while new product pages sit undiscovered for weeks. Blocking those filter paths in robots.txt is not optional on sites like that. It is how you ensure Googlebot actually reaches your inventory.

A correct robots.txt file

Here is what a well-structured robots.txt looks like for a typical e-commerce site. Note the Allow rule carving an exception out of the broader Disallow, and the Sitemap directive at the bottom:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/
Disallow: /filter/
Allow: /filter/featured/

User-agent: Googlebot
Disallow: /staging/

Sitemap: https://www.example.com/sitemap.xml

The Allow: /filter/featured/ line demonstrates path-specific exceptions. If your filter system generates one curated landing page you actually want indexed, an Allow rule with a more specific path takes precedence over the broader Disallow covering the rest of the filter directory.

Syntax that trips people up

The format is simple but unforgiving. A single stray rule can wipe out an entire section of a site, and because it ships as a deploy artifact it tends to fail quietly until traffic drops.

  • User-agent targets a specific crawler, or * for all of them at once.
  • Disallow: / blocks the whole site. I have seen this reach production after a launch when nobody remembered to remove the staging rule.
  • Matching is path-prefix based, and an Allow line can carve a specific exception out of a broader Disallow.
  • Each subdomain needs its own robots.txt. The file on your apex domain does not cover a shop. or blog. subdomain.

Precedence in robots.txt is determined by the length of the matching rule, not by the order in which rules appear in the file. The most specific rule wins. This catches people who assume that a later Allow rule overrides an earlier Disallow the way CSS cascade works. It does not. A shorter Disallow will lose to a longer Allow, regardless of file order.

The mistake to avoid

Do not block a page in robots.txt and also expect a noindex tag on it to work. If the crawler cannot fetch the page, it never sees the noindex, so the page can linger in the index indefinitely, often with the unhelpful note that no information is available. To reliably remove a page, you have to let it be crawled and serve a noindex instead. Crawling and indexing are two separate gates, and robots.txt only touches the first one.

This is also why blocking JavaScript and CSS files in robots.txt is a problem for JavaScript-heavy sites specifically. If Googlebot cannot fetch the script files it needs to render a page, it cannot produce a rendered version of that page for the index. The result is pages that are technically crawled but arrive in the index as near-empty documents, often triggering "Crawled but not indexed" status in Google Search Console. Unblock those files and the rendering problem frequently resolves on its own.

How to test a robots.txt file

Before I trust a robots.txt file, I test it against real URLs rather than reading it top to bottom and hoping. The interaction between competing rules is where the surprises hide, and a rule that looks harmless can override another you assumed was winning.

  • Pull a sample of important URLs and confirm each one is allowed, not just the homepage.
  • Check that the most specific matching rule wins, because precedence is decided by rule length, not by the order lines appear in the file.
  • Watch for a trailing rule or a comment that accidentally extends a Disallow path further than intended.
  • Re-test after every deploy, since this file is generated by the same pipeline that can quietly break it.

Google Search Console has a robots.txt tester built into the Legacy Tools section. It is not prominently featured anymore, but it is still accessible and genuinely useful because it tests against the live file at your domain rather than a local copy that may differ from what is deployed. Screaming Frog also has a robots.txt checker that will run a list of URLs against a specified robots.txt and report which ones would be blocked, which is faster than testing URLs one at a time through a web interface.

Robots.txt and your XML sitemap

The Sitemap directive in robots.txt is worth using even though it is not technically required. Many crawlers read robots.txt before anything else, and a Sitemap line there means they discover your sitemap immediately rather than waiting to encounter a link to it. It also gives you one canonical place to declare where your sitemap lives, which is useful if you run multiple crawlers or use third-party indexing services.

One consistency check worth running: compare the URLs in your sitemap against your robots.txt rules. A URL that appears in your sitemap but is also blocked by a Disallow rule sends a contradictory signal. Google will typically crawl the URL anyway because sitemaps are high-priority, but it creates unnecessary confusion in coverage reports and is a sign the two files are not being maintained together.

That crawling-versus-indexing line is exactly where most accidental deindexing and accidental indexing happens, so it is worth understanding both sides clearly. I break down the indexing controls in noindex vs nofollow. If you want me to check whether your robots.txt is quietly blocking pages it should not, that is a core part of a website audit.