AI crawler policy, re-measured after a parser fix (October 2026)
Of 120,190 websites whose robots.txt has a rule that reaches AI crawlers, OpenAI's model-training crawler GPTBot was blocked on 10.39% and its search crawler OAI-SearchBot on 5.48%. Counted across all 162,649 sites in this post-fix sample, including those with no rule that reaches AI crawlers, the rates are 7.68% and 4.05%.
About this sample. Post-fix audits only: the latest audit per domain on or after 26 August 2026. These audits come from our own scanning campaigns, so the sample is not random and is not the whole SearchScore corpus.
This study and the 18 August 2026 study use different samples and different denominators, so the change between them is not a trend and must not be presented as one. The headline here counts only the 120,190 sites whose robots.txt has a rule that reaches AI crawlers, because a site with no robots.txt, or one whose rules do not reach AI crawlers, gives us no rule to read. The all-sites figure, over 162,649 sites, is given alongside. The August study divided by every unique domain audited in its 90-day window.
OpenAI documents the two crawlers separately. GPTBot crawls content that may be used to train its generative AI foundation models. OAI-SearchBot makes content eligible to be discovered and cited in ChatGPT search. A site can block one and leave the other open, and 5,782 of the 12,490 GPTBot blockers in this sample do.
"Allowed" here means the crawler is not blocked and a rule reaches it. That rule can be a group naming the crawler or the wildcard (*) group, so it does not mean the site named OAI-SearchBot.
Why we measured again
On 25 August 2026 we fixed a bug in the robots.txt parser our audits use. Rules from a later user-agent group leaked onto earlier agents, so a file that allowed OAI-SearchBot and then disallowed GPTBot read OAI-SearchBot as blocked. A Content-Signal: line also cut the wildcard group off from its rules. The 18 August study was measured before the fix, and it stays published with its original figures and a dated note.
This re-measure uses only audits taken after the fix, from 26 August 2026 onwards, so every robots.txt in it was read by the corrected parser.
All measured tokens
The wider table is context, not a ranking of training against retrieval crawlers. The roles are not uniform: Google-Extended is a robots control token covering both Gemini training and grounding, not a separate HTTP crawler, and CCBot builds Common Crawl's general open-web dataset.
| Token | Conservative role label | Blocked | With a rule | All sites (post-fix sample) |
|---|---|---|---|---|
| GPTBot | OpenAI model-training crawler | 12490 | 10.39% | 7.68% |
| ClaudeBot | Anthropic model-development/training crawler | 11614 | 9.66% | 7.14% |
| Google-Extended | Google robots control token for Gemini training and grounding | 10942 | 9.1% | 6.73% |
| CCBot | Common Crawl open-web dataset crawler | 10285 | 8.56% | 6.32% |
| Bytespider | ByteDance crawler; exact public purpose not relied on in the headline | 10104 | 8.41% | 6.21% |
| PerplexityBot | Perplexity search-indexing crawler | 7240 | 6.02% | 4.45% |
| OAI-SearchBot | OpenAI search crawler | 6583 | 5.48% | 4.05% |
| GoogleOther | Google generic crawler used by its product teams; role not relied on in the headline | 6472 | 5.38% | 3.98% |
| Claude-SearchBot | Anthropic search crawler | 6260 | 5.21% | 3.85% |
| DeepSeekBot | Other measured crawler token; role not relied on in the headline | 6220 | 5.18% | 3.82% |
"With a rule" divides by the 120,190 sites whose robots.txt has a rule that reaches AI crawlers, by name or through the wildcard (*) group; "All sites (post-fix sample)" divides by 162,649. The blocked count is the same in both, because a site with no such rule blocks none of them.
What this does not prove. This is a snapshot of one sample, not a random sample of the web and not a time series. Robots policy does not tell us why a rule was configured, whether the crawler obeyed it, whether content was used for training, or whether the site later appeared in an AI answer.
Methodology
We took every domain whose latest SearchScore audit is on or after 26 August 2026: 164,635 domains. We excluded the 1,986 whose robots.txt could not be reached, leaving 162,649. That includes 3,112 audits that recorded no robots.txt status at all; they are counted using the crawler verdicts each of those audits did record, the same as every other site. Of the 162,649, 120,190 have a robots.txt that blocks or allows at least one measured AI crawler, by name or through the wildcard group. The other 42,459 have no robots.txt, or none that reaches AI crawlers.
The audits come from our own scanning campaigns. Which domains we scan, and when, follows our campaigns rather than a sampling frame, so these figures describe the sites in this sample and not the web as a whole.
For the wider table we use conservative labels. Anthropic says ClaudeBot collects public web content that could contribute to model training. Perplexity says PerplexityBot is for surfacing and linking sites in search results, not foundation-model training. Google says Google-Extended is a control token for both Gemini training and grounding. Common Crawl describes CCBot as the crawler that builds its open web archive. We do not rely on an exact Bytespider, GoogleOther or DeepSeekBot role to support the headline.
- OpenAI publisher and search-crawler guidance
- Anthropic crawler guidance
- Perplexity crawler guidance
- Google crawler and Google-Extended guidance
- Common Crawl CCBot documentation
Download and cite the data
The aggregate data behind the findings is available for reuse. Row-level domain data is not published.
For a sector or country-specific cut of this dataset, contact [email protected].