Blocking AI Crawlers: A Policy-First robots.txt Guide

TL;DR: There is no single "block AI" switch. Decide policy per purpose (training, search, user fetch) and per content class first, then write narrow robots.txt rules and verify them at three layers.

"Block AI" sounds like a single toggle. It isn't. A publisher may want to restrict model training, stay eligible for AI-search citations, and still let an agent fetch a page when a user explicitly asks. Those are different purposes, governed by different user agents and controls.

The wrong robots.txt rule can create a false sense of protection, or quietly remove content from a discovery channel the business intended to measure. The right answer starts with policy, not syntax.

Key Takeaways

"Block AI" is four separate decisions, not one: training use, search indexing, user-triggered access, and referral/attribution. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as independent controls, so you can allow search visibility while disallowing training.
Robots.txt is an access-policy request to compliant crawlers, not a contract. It doesn't authenticate visitors, stop already-copied content, remove third-party references or create a licence. Confidential material needs real authentication and server-side access control.
Google-Extended is a product token for certain Gemini uses, not a separate HTTP crawler, and Google says it does not affect inclusion or ranking in Google Search or remove pages from AI Overviews.
User-agent strings are trivially spoofed. Verify bots using each provider's published IP guidance before trusting logs, and never allow broad network ranges from unofficial lists without security review.
Access does not guarantee retrieval, citation, recommendation or a lead. Measure presence and commercial results separately, and review policy quarterly rather than building it from an outdated bot list.

What are you actually trying to block? Separate four questions

Before touching a file, split the decision into four questions and ask them of each provider:

Paper-cut website gate separating AI crawlers by approved purpose

Training: may content be used to improve foundation models?
Search indexing: may an automated bot collect content for search or answer retrieval?
User-triggered access: may an agent fetch a URL because a user explicitly requested it?
Referral and attribution: can resulting visits or citations be observed?

OpenAI's official documentation distinguishes GPTBot, OAI-SearchBot and ChatGPT-User, and states that their controls are independent. A publisher can, for example, allow OAI-SearchBot for search visibility while disallowing GPTBot. Perplexity likewise documents separate crawler and user-agent purposes. Do not generalise one provider's model to another.

Does robots.txt actually protect anything? No, it's access policy, not a contract

Robots.txt is a public, machine-readable request to compliant crawlers, and nothing more. It does not authenticate a visitor, stop copied content already obtained elsewhere, remove third-party references, create a licence agreement or prevent a malicious scraper. Disallowed URLs may sometimes still be known as URLs without their content being crawled.

For confidential or paid material, use real access control: authentication, authorisation and appropriate delivery architecture. Never put secrets in a public file and expect Disallow to protect them.

Google's documentation adds an important distinction. Google-Extended is a product token governing certain uses by Gemini models; it is not a separate HTTP crawler, and Google says it does not affect inclusion or ranking in Google Search. Controls for Search features must be understood through Google's Search documentation rather than inferred from the word "Extended".

How do you keep track of every crawler? Build a register

Maintain one record so policy never goes orphaned, with these fields:

Field	Why it matters
provider and exact token	avoids rules aimed at invented bot names
documented purpose	training, index, user fetch or other
official source and review date	controls change
desired policy by content class	public blog may differ from licensed archive
robots rule	implementation evidence
edge/CDN rule	may override or contradict robots
verified traffic	confirms observed requests, not identity by name alone
owner and next review	prevents orphaned policy

A user-agent string is trivially spoofed. Where a provider publishes IP verification guidance, validate using that method before treating logs as genuine. Do not allow broad network ranges copied from an unofficial list without security review.

Analyst reviewing crawler access signals in a midnight operations room

Should the same rule cover every page? No, decide by content class

Policy should follow the content, not a blanket setting. Work through it class by class.

Public commercial pages

If the business wants discovery and citations, allowing documented search crawlers can be coherent. Measure the result: access does not guarantee retrieval, citation, recommendation or a lead.

Original research

The choice is strategic. Discoverability may build authority, but licensing or competitive concerns may justify restrictions. Align legal, editorial and commercial owners rather than allowing SEO to decide alone.

Licensed, personal or confidential content

Robots.txt is insufficient here. Confirm contractual rights and enforce server-side access. Avoid exposing sensitive content to any unauthorised requester, human or automated.

UGC and mixed-rights pages

The site may own the platform but not every reuse right. Review terms, consent, moderation and jurisdiction with qualified counsel. This article is operational guidance, not legal advice.

What should the rules actually look like? Write them narrow

Keep the ruleset as small as the policy allows. A simple policy may look like this conceptually:

```text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

Overhead robots policy cards with allow and block symbols

Do not paste that into production merely because it appears here. Check current official token names, existing wildcard groups, sitemap declarations, generated environments and CDN behaviour. A broad User-agent: * block elsewhere may alter the outcome, and some frameworks regenerate robots.txt during deployment.

Stage the change, test the live file over HTTPS, retain the previous version and record approval. A syntax check alone is not acceptance.

How do you know it worked? Verify at three layers

A change isn't done until it's confirmed at configuration, request behaviour and business observation.

1. Configuration

Fetch the public robots.txt from multiple environments. Confirm status code, content type, redirects and that production, not a cache or preview, is being inspected.

2. Request behaviour

Use controlled test paths where appropriate. Review origin and CDN logs. Distinguish compliant absence from a logging gap. Check that security tools are not blocking the allowed bot or allowing a disallowed one independently.

3. Business observation

Track stable prompt samples, visible citations and referrals separately. A before/after citation screenshot does not prove the robots change caused the outcome; AI interfaces vary and discovery can lag.

SAGEO's AI search dashboard framework separates presence from commercial results. Its citation-without-attribution analysis explains why even a visible source is not automatically brand memory. Both sit within how SAGEO defines the discipline of SEO, AEO and GEO together.

Where do teams get this wrong? Common failures

The recurring mistakes are predictable:

Glass data greenhouse with selectively open crawler gates

blocking every token containing "AI" without mapping purpose;
assuming GPTBot controls ChatGPT search;
assuming Google-Extended removes a page from AI Overviews;
using robots.txt for confidential content;
trusting user-agent strings without published verification;
changing CDN bot settings but not documenting them;
letting a plugin overwrite rules on deployment;
counting a fall in bot hits as proof content is no longer used; and
promising citations after access is opened.

What does a sensible rollout look like? A defensible 30-day plan

Move deliberately, one week at a time:

Week 1: inventory content rights, current robots rules, edge controls and observed crawlers.

Week 2: agree policy by purpose and content class with legal, security, editorial and growth owners.

Week 3: implement the smallest ruleset, stage, test and retain rollback evidence.

Week 4: monitor verified requests, indexing and a fixed commercial prompt set. Report uncertainty and exceptions.

Review quarterly, or whenever a provider changes documentation. Do not build a permanent policy from a 2024 bot list.

The decision rule

Allowing a crawler is sensible when its documented purpose supports an approved business goal and the content is safe and permitted to expose. Blocking is sensible when that use conflicts with rights, policy or risk appetite. Authentication is mandatory when content must actually remain private.

Cross-functional team governing AI crawler policy around a round table

Need an auditable policy rather than a copied blocklist? Request a SAGEO technical visibility audit covering robots.txt, edge controls, verified bot traffic and business measurement.

Sources

OpenAI, crawler documentation
OpenAI, publishers and developers FAQ
Google, common crawlers and Google-Extended
Google, AI features and your website
Perplexity, crawler documentation