More than half the sites that block ChatGPT's training crawler have also blocked themselves out of its answers.

RBy Ronnie Huss, founder of SearchScore

A training crawler and a search crawler do different jobs, and the distinction is load-bearing.

OpenAI's documentation is explicit. GPTBot "is used to make our generative AI foundation models more useful and safe". OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and sites opted out of it "will not be shown in ChatGPT search answers". Anthropic splits the same way, with ClaudeBot collecting content that may contribute to training and Claude-SearchBot indexing for search.

So a site can refuse training and keep the answers. We wanted to know how many actually do, so we measured it.

What we found

We read the robots.txt directives captured at audit time for every site in the corpus with a recorded AI crawler policy, which is about two thirds of them, on 26 August 2026. Figures below are shares of that group.

Roughly one site in six now refuses AI training, and the three training crawlers are blocked at almost identical rates. That part is unsurprising and widely reported.

The overlap is where it gets interesting.

More than half of them lose the answers as well

Of the sites that block GPTBot, 56.2% also block OAI-SearchBot. They have refused training and, in the same file, removed themselves from ChatGPT's search answers.

7.5% block GPTBot and leave OAI-SearchBot open. That is the configuration you would choose deliberately, and roughly two in five of the sites refusing OpenAI training have arrived at it.

We went in expecting the opposite error, sites keeping training while losing the answers by accident. It is almost nonexistent: 0.07%, about one site in 1,400. Whatever is producing the 56.2% overlap, it is not people deliberately choosing to be trained on while declining to be cited.

The Anthropic number tells you what is happening

16.3% of sites block ClaudeBot. Fewer than one in 50,000 block Claude-SearchBot. A ratio of roughly nine and a half thousand to one.

That cannot be a considered position held independently by tens of thousands of site owners. It is what a copy-pasted robots.txt block looks like at scale. The lists that circulate name GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, because those were the names that existed when the lists were written. OAI-SearchBot has since been added to some of them. Claude-SearchBot has been added to almost none.

Read that way, the OpenAI overlap is the same artefact one revision later: a list that got updated, applied by people who never revisited what the new line did.

The obvious objection, tested

There is a reading of the 56.2% that does not need a copy-paste explanation at all. Plenty of publishers decided in 2023 and 2024 that they wanted nothing to do with AI, and blocking both crawlers is exactly what that decision looks like. If that were the driver, the sites blocking both would skew towards media and publishing.

So we checked. Comparing the sector mix of the sites blocking both OpenAI crawlers against those blocking GPTBot alone, media sites are 1.96% of the first group and 2.10% of the second. Publishers are marginally less likely to be in the block-both group, not more. Across every sector large enough to read, the two groups have close to the same composition.

That is the opposite of what a deliberate, sector-specific opt-out would produce, and it is what you would expect if the difference between the two groups is which robots.txt template somebody copied.

Worth stating the limit: most of the corpus carries no sector classification, so this compares the classified remainder. It makes the copy-paste reading more likely without settling it, and robots.txt alone cannot record why a rule was written.

What this means if you are one of them

Open your robots.txt and search it for "SearchBot". If the string does not appear, you have never made this decision, and on these numbers there is a better than even chance you are blocked out of answers you would like to be in.

The configuration that refuses training and keeps answers looks like this.

Allow all four if you are relaxed about training. The point is that it is a choice, and right now more than half the people who made one got the other as well.

What this measurement is and is not

It reads robots.txt directives captured when we audited each site, so it describes stated policy rather than crawler behaviour. It says nothing about whether the crawlers honour those rules, and nothing about whether a site that permits a crawler is actually cited.

Sites with no robots.txt, or with no AI crawler rules at all, are excluded from the denominator rather than counted as permitting everything. About a third of the corpus falls into that group. Including them would roughly halve every share above and would confuse a deliberate decision with the absence of one.

Ronnie has written up what to do about it, and the advice he had to correct on his own site.