Robots.txt: Block Crawling, Not Indexing

By Firdaus Nagree

TL;DR: Robots.txt controls crawler access to URL paths, but it is not a reliable way to keep a web page out of Google Search.

Key takeaways

  • Robots.txt manages crawling rather than guaranteeing removal from an index.
  • A blocked URL can still appear in search results if Google learns about it elsewhere.
  • Google must crawl a page to read a page-level noindex instruction.
  • Rules apply to the host, protocol and port where the file lives.
  • Test priority URLs and keep a rollback copy before changing live rules.

What does robots.txt actually control?

Robots.txt tells compliant crawlers which URL paths they may access on a particular site. Google describes it mainly as a way to manage crawler traffic, not as a mechanism for keeping a web page out of Search.

The file belongs at the root of the relevant host. Google's robots.txt specification explains that rules apply only to the same host, protocol and port. A rule on one subdomain does not quietly govern every other subdomain because the internet declined to make configuration that forgiving.

Abstract website gate controlling crawler paths

Robots.txt is public. Do not place secrets in it or treat a disallow rule as access control.

Can robots.txt stop a page being indexed?

No, robots.txt cannot reliably stop a page URL being indexed. Google's introduction to robots.txt says a blocked page can still appear in results when other pages link to it, although Google may not have content for a useful snippet.

If the goal is to keep a page out of Google, choose a method suited to that goal. Google points to noindex for index control and password protection for private content.

The distinction matters during migrations, staging work and faceted navigation projects. A crawl block may reduce access while leaving an already known URL eligible to appear as a bare result.

Why can robots.txt and noindex conflict?

A robots.txt block can prevent Google from fetching the page that contains a noindex rule. Google's robots meta tag guidance states that page-level rules can be read only when crawlers are allowed to access the page.

Crawler figurine beside a plain configuration sheet and URL cards

This creates a common trap. A team adds noindex and disallow together, expecting two layers of exclusion. The crawl block can stop the crawler seeing the noindex instruction at all.

Audit the current state before prescribing the fix. Record whether the URL is crawlable, whether it returns the intended status, whether a noindex instruction is present in the rendered response and whether the page is already known in Search Console.

Which directives does Google support?

Google supports user-agent, allow, disallow and sitemap fields in robots.txt. Its specification says crawl-delay is not supported by Google.

Matching is based on URL paths, and rule specificity matters. Google says the most specific matching path wins, while equally specific conflicting rules use the least restrictive rule.

Do not review rules as isolated lines. Grouping, wildcards, end markers, case and percent-encoded paths can change the result. Test representative URLs from every important template rather than admiring the syntax in a document.

Which mistakes cause the most damage?

The most damaging mistakes are broad rules, environment leakage and untested deployment changes. A single disallow rule can affect product pages, articles, scripts or style resources if its path matches more than intended.

Crawling path and indexing catalogue shown as separate systems

Common audit findings include:

  • a staging rule copied into production;
  • a path that differs only by letter case;
  • important CSS or JavaScript resources blocked from rendering;
  • separate rules on one host assumed to cover another;
  • a noindex instruction hidden behind a crawl block;
  • a sitemap URL pointing to the wrong host or environment.

SAGEO's internal-links audit guide is useful alongside this work because crawler access and internal discovery are separate questions. Fixing one does not automatically fix the other.

How should you audit a live robots.txt file?

Audit the served file, not only the repository copy. Fetch the exact production URL, record its status and response body, then compare it with the intended configuration.

Build a small test set containing the homepage, key commercial templates, articles, assets, faceted URLs and deliberately blocked areas. For each URL, record the expected crawler, expected access decision, actual rule and business reason.

Technical workspace for checking logs and crawler responses

Review server logs or crawl evidence after changes where available. A testing tool can confirm rule interpretation, but natural crawling shows whether the live system and internal links are behaving as expected.

What should happen before a change goes live?

Before a change goes live, save the current file, peer-review the exact replacement and test it against the priority URL set. Define rollback and the post-deployment checks in advance.

Check that the file remains reachable and contains only the intended rules. Then verify priority pages, rendered resources and affected crawl patterns. SAGEO's XML sitemap guide explains how sitemap discovery should be checked separately from crawl permissions.

A successful upload is not acceptance. It is merely the moment the test becomes interesting.

Limitations

Crawler behaviour differs between services, and robots.txt is voluntary for crawlers outside the systems described by their documentation. Google may update its processing rules and tools. Search Console and log data can lag, and neither proves that every crawler saw the same response at the same moment.

Miniature website launch test route with rollback box

What is the next sensible step?

Export the live robots.txt file, list the URL groups it is meant to control and test real examples before editing. Ask SAGEO for a crawl-control audit when several hosts, templates or deployment systems make the blast radius difficult to judge.

Sources