Robots.txt: Control Crawling, Not Indexing

By Firdaus Nagree

TL;DR: Robots.txt controls crawler access to URL paths, but it is not a reliable way to remove a web page from Google's search results.

Key takeaways

  • Robots.txt manages crawling, not guaranteed indexing or removal.
  • A blocked URL can still appear in search if Google discovers it elsewhere.
  • Crawlers must access a page to see its robots meta tag or HTTP indexing directive.
  • Test rules against representative URLs before changing production.
  • Keep a versioned backup, monitor crawler responses and prepare a fast rollback.

What does robots.txt do?

Robots.txt tells compliant crawlers which URL paths they may access. Google says its main purpose is managing crawler traffic and preventing unnecessary requests, not keeping ordinary web pages out of Search: Google's robots.txt introduction.

The file normally sits at the root of a host, such as https://example.com/robots.txt. Rules are scoped to that protocol, host and port. A rule on one subdomain does not quietly govern every other subdomain because it feels like part of the same brand.

Treat the file as public configuration. Anyone can request it, so it must not reveal secret paths or become a substitute for authentication.

Luminous website gate controlling crawler paths on a network

A blocked URL can still appear because Google may discover it through links or other signals without crawling its content. Google explicitly warns that a robots.txt block can leave the URL indexed without a descriptive snippet.

That distinction is the centre of a safe implementation: discovery, crawling and indexing are related but separate. Blocking the crawl does not erase knowledge of the URL. Robots.txt is a gate, not a memory wipe wearing a technical hat.

If the objective is removal from search, choose a method designed for indexing control and ensure the crawler can see it. For private material, require authentication rather than relying on a publicly visible directive.

When should you use noindex instead?

Use noindex when an accessible page should not appear in Google Search. Google's robots meta tag documentation says a robots meta tag can control indexing for an HTML page, while an X-Robots-Tag HTTP header can apply equivalent rules to non-HTML resources such as PDFs.

The crawler must be allowed to fetch the URL to read either instruction. Combining a robots.txt block with an unseen noindex can therefore frustrate the intended result. Google cannot follow an instruction hidden behind the gate you just closed.

Authentication is the stronger choice for confidential or user-specific content. A search directive is not access control, and a URL omitted from results may still be reachable to anyone who has it.

Allow and block tokens arranged with website cards and a crawler model

Which URLs are suitable for crawl blocking?

Crawl blocking is most suitable for URL patterns that create unnecessary crawler load and have no useful search destination, provided the consequences have been tested. Examples may include repetitive filtered states, internal search combinations or unimportant generated paths.

Do not copy generic directives into production. A pattern that is harmless on one platform can remove access to products, localisation files, images, scripts or rendered content on another. Classify each candidate pattern by template, business value, internal links, current indexing and resource dependencies.

Google warns against blocking resources that it needs to understand a page. If CSS or JavaScript materially changes the rendered content, test how Google sees representative pages before and after the proposed rule.

How do allow and disallow rules work?

Allow and disallow rules apply to the user-agent groups and path patterns defined in the file. The effective result depends on rule specificity and the crawler's implementation, so tests should use the exact production-style URL rather than a simplified example.

Preserve query strings, trailing slashes, case and encoded characters in the test set. Include one URL expected to be allowed and one expected to be blocked for every changed pattern. Add edge cases from live logs rather than relying only on what the CMS says it can generate.

Comments can explain ownership and intent, but they should not contain sensitive information. Keep them concise and update them when the rule changes. A six-year-old comment describing a platform that no longer exists is historical fiction, not operational documentation.

Blocked crawl gate beside a URL still visible in an index library

Should the file include a sitemap location?

A robots.txt file can declare one or more sitemap locations, which can help crawlers discover the submitted files. The sitemap must still contain preferred, indexable URLs and return a valid response.

A sitemap declaration does not override a crawl block, canonical conflict, noindex directive or error status. Align the systems rather than asking one line in robots.txt to mediate an argument among five templates.

SAGEO's XML sitemap guide explains the separate discovery role of sitemaps. Internal links still matter too. The verified internal-links audit guide covers crawlable anchors and destination checks.

How should you audit an existing robots.txt file?

Audit the live file, historical versions, crawler logs and representative URLs together. Start by recording the final robots.txt URL, response status, content type, body, redirect behaviour and cache headers for every important host.

Map each user-agent group and directive to the paths it affects. Then sample high-value pages, pagination, filters, internal search, media, scripts, stylesheets, APIs and genuine error URLs. Compare intended access with observed requests in server logs.

Check whether staging rules have leaked into production or production rules have been copied into staging without access control. A blanket block on staging is useful only if staging is also properly protected from unauthorised visitors.

Server operations room monitoring crawler request paths

What should a safe change plan include?

A safe change plan names the exact host and file, current content, intended line-level edit, affected patterns, test URLs, owner, deployment time and rollback. Export the previous file before changing it.

Test syntax and representative matches in the available search-engine tools, then fetch the deployed file directly. Verify allowed and blocked URLs separately. Watch server logs, Search Console crawl information and index coverage for unexpected changes over an appropriate period.

Change one coherent rule set at a time. Replacing the whole file during a platform migration makes attribution needlessly difficult if crawl behaviour shifts.

Does robots.txt control AI crawlers too?

Robots.txt can express policies for crawlers that document and honour relevant user-agent rules, but crawler identities and purposes differ. Separate search crawling, AI answer retrieval, model training and user-triggered access in the policy discussion.

Do not assume that one broad user-agent block matches the organisation's legal, commercial or discoverability goals. Record which crawler is being addressed, why, what paths are affected and how compliance will be monitored.

Limitations

Robots.txt is a voluntary crawl-control protocol, not access control, guaranteed deindexing or a ranking tool. Behaviour can differ among crawlers. Search results, log patterns and Search Console reports may take time to change, and no implementation guarantees rankings, traffic or answer-engine citations.

Version branch, deployment cards and rollback control in a studio

What is the next sensible step?

Fetch every production host's live robots.txt file, classify each rule and test a representative allowed and blocked URL. Ask SAGEO for a crawl-control audit if the file spans multiple platforms, subdomains or generated URL systems.

Sources