AI bot permissions in robots.txt: The complete guide

Your robots.txt file can affect which automated systems may crawl your website, but AI providers use different tokens for search, training, grounding and user-triggered fetches. This guide separates those roles so you can protect content without accidentally closing a search-discovery path.

Key Takeaway

Do not treat every AI-related token as the same kind of crawler. OpenAI separates OAI-SearchBot (search discovery) from GPTBot (training); Anthropic documents separate search and training agents; PerplexityBot is a search crawler; Google-Extended is a product-control token rather than the crawler that governs Google Search or AI Overviews. Decide each role deliberately.

Why this is the most important GEO fix

A robots.txt block is urgent when it affects a discovery crawler the engine actually relies on. If OAI-SearchBot cannot access a public page, for example, OpenAI says its content may not appear in ChatGPT search summaries and snippets. That is an eligibility problem, not proof that robots.txt is the biggest visibility issue on every site.

There is one distinction to get right before you write a single line: provider tokens have different jobs. OpenAI uses GPTBot for training-oriented crawling and OAI-SearchBot for search discovery. Anthropic documents separate training and search agents. PerplexityBot is used for search. Google is different again: Googlebot underpins Search and Search AI features, while Google-Extended controls certain Gemini training and grounding uses. Classify the role before deciding whether to allow or block it.

SearchScore’s interactive audit data (6,944 websites, July 2026) shows 6.9% of websites (roughly 1 in 14) block at least one major AI crawler. The vast majority do this accidentally - through legacy robots.txt rules written before AI search existed.

Check which AI crawlers can access your site

Enter your domain to see how OAI-SearchBot, Claude-SearchBot, PerplexityBot and other relevant agents are treated. Review Google-Extended separately as a training/grounding control rather than as a Search crawler.

Check

The most common accidental AI block

The most frequent culprit is a blanket disallow rule:

User-agent: *
Disallow: /

This tells every bot - including all AI crawlers - that they cannot access any page on your site. It is often added to staging or development sites and accidentally left in place, or added to “protect” a site from spam bots without realising it blocks everything.

The major AI crawler user-agents

Sorted by what each one actually decides. If you only act on one row, make it a search crawler.

Two user-agents you may still see recommended elsewhere, anthropic-ai and Claude-Web, no longer appear in Anthropic’s published crawler documentation. Leaving them in your file is harmless, but they are not what governs Claude access today.

The recommended robots.txt configuration

A visibility-oriented policy might allow the documented search/discovery crawlers while making separate choices about training and grounding controls:

# Search/discovery crawlers - relevant to provider search eligibility
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Google Search / AI Overviews use Googlebot. Google-Extended is not a Search crawler.
# Decide Google-Extended separately if you want to control Gemini training/grounding.

# User-initiated fetches - someone asked the assistant about your page
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Training crawlers - allow or disallow as a policy choice.
# Blocking these does NOT remove you from AI search.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Your existing rules below
User-agent: *
Disallow: /wp-admin/
Disallow: /private/

Separating search retrieval from training data

This is the most common thing site owners want, and the most common thing they get backwards: allow AI search retrieval so you can be cited, while declining to have your content used for model training. It is entirely achievable, because both major providers give you a separate user-agent for each.

For OpenAI, allow OAI-SearchBot if you want public content eligible for ChatGPT search discovery and make the GPTBot training decision separately. OpenAI’s current publisher guidance says sites should not block OAI-SearchBot if they want content included in ChatGPT search summaries and snippets.

For Anthropic, allow Claude-SearchBot and disallow ClaudeBot. Anthropic documents ClaudeBot as collecting content that “could potentially contribute to their training”, and Claude-SearchBot as what indexes your content for search.

# Cite me, but do not train on me
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

The point is not that every provider behaves identically; it is that search eligibility and training permission can be different controls. Check each provider’s current documentation before you deploy, because user agents and policies change.

How to test your configuration

1. Visit yoursite.com/robots.txt and review the rules

2. Request a page with the crawler’s user-agent set and check you get a 200, for example curl -A "OAI-SearchBot" -I https://yoursite.com/. Search Console’s robots.txt report shows you the file Google fetched, but it cannot test other providers’ user-agents

3. Re-test after the provider has had time to refetch robots.txt; providers do not publish one universal cache or recrawl timetable, so avoid promising a fixed 24-48 hour effect

4. Run a SearchScore audit to verify AI citability signals are now passing

Back to pillar

- Technical GEO: How to Optimise Your Website for AI Search →

S

Ronnie Huss

GEO Research & Analysis

The SearchScore editorial team researches and writes about generative engine optimisation, AI search visibility and the signals that determine whether your website gets cited by ChatGPT, Perplexity and Google AI Overviews.

Sources & Further Reading

- OpenAI – OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User

- Anthropic – Does Anthropic crawl data from the web? (ClaudeBot)

- Perplexity – PerplexityBot crawler documentation

- Google Search Central – Overview of Google crawlers and user agents

Frequently Asked Questions

What are the main AI crawler user-agent names?

They split into two groups. Search crawlers, which decide whether you can be cited: OAI-SearchBot (OpenAI/ChatGPT search), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Googlebot (Google, including the index AI Overviews cites from) and Bingbot (Microsoft Copilot). Training crawlers, which do not affect search visibility: GPTBot (OpenAI) and ClaudeBot (Anthropic). Google-Extended is a separate control for Gemini grounding and training. ChatGPT-User and Claude-User handle fetches a person triggered by asking about your page.

How do I check if my robots.txt is blocking AI crawlers?

Visit yoursite.com/robots.txt and look for any User-agent: * rule with Disallow: / - this blocks all crawlers including AI bots. Then check for explicit blocks on the search crawlers specifically: OAI-SearchBot, Claude-SearchBot and PerplexityBot. To verify one directly, request a page with that user-agent set, for example curl -A “OAI-SearchBot” -I https://yoursite.com/.

Should I allow AI crawlers for training data collection?

This is a policy choice. Where a provider exposes separate search and training agents, you can allow the search/discovery path while blocking the training crawler if that matches your policy. Do not generalise that pattern to Google-Extended: Google Search and AI Overviews use Googlebot, while Google-Extended is a separate training/grounding control.

Check your AI visibility

Enter your URL at SearchScore for a free AI visibility Score. See how ChatGPT, Perplexity and Google AI see your site - and exactly what to fix.

See whether your fixes worked

The Tracker re-checks your site every week and shows which scores moved. Start with the free scan, then open a Founders Tracker account from your fix list. No card.

Run my free scan →