▓Benchmark GEO & AI Search

6.9% of websites block at least one major AI crawler

Roughly 1 in 14 sites block a major AI crawler such as GPTBot, ClaudeBot or PerplexityBot in their robots.txt, usually by accident.

ID
SS-BM-001
Confidence
High · 92
Evidence
Strong
Updated
2026-07-08
6.9% Roughly 1 in 14 sites block a major AI crawler such as GPTBot, ClaudeBot or PerplexityBot in their robots.txt, usually by accident.
Sample
~6,944 websites
Period
July 2026
Source
SearchScore interactive audit data
Updates
Auto-refreshed

What this means

A blocked crawler removes a site from the live AI answer layer today and from future training snapshots, and no Google-only rank tracker will ever surface it. The block is almost always inadvertent, caused by legacy wildcard rules or bot filters that predate AI search. It is one of the cheapest problems to fix and the most damaging to leave in place.

How do you compare?

Recommended next steps

    Apply the method Crawlability Framework Framework → See the wider capability Crawl Optimisation Capability →