Executive summary
AI crawlers separate into training crawlers, which harvest content for model training, and answer crawlers, which fetch pages to cite in live responses. Blocking training crawlers protects proprietary content with little visibility cost; blocking answer crawlers can remove you from AI results entirely. Decide per bot, not with a blanket rule.
What this helps you decide
Whether allowing a given AI crawler advances your visibility more than it risks your content.
Business problem
AI crawlers now fetch content to train models and to answer live queries. Blocking them can protect content but can also make your brand invisible in AI answers; allowing them without thought can expose material you would rather keep out.
Step-by-step process
-
1
Separate training from retrieval bots
Distinguish crawlers that gather training data, such as GPTBot and Google-Extended, from retrieval agents that fetch live pages to cite, such as answer-time user agents.
-
2
Decide your default posture
For most publishers seeking AI visibility, allow retrieval crawlers and make the training decision separately based on how proprietary your content is.
-
3
Write explicit user-agent groups
Add named blocks for each bot rather than relying on the wildcard, because AI crawlers only obey rules that match their exact user-agent token.
-
4
Scope allow and disallow paths
Permit your public, citable content while disallowing account areas, checkout and anything you never want quoted.
-
5
Keep search-engine access intact
Never let an AI-crawler rule accidentally block Googlebot or Bingbot; test that your classic search crawlers remain fully allowed.
-
6
Validate and publish
Test the file in a robots checker, confirm each user-agent resolves the intended rules, then publish at the domain root.
-
7
Monitor in logs
Watch server logs to confirm each bot honours its rules and to spot new AI user-agents worth a decision.
Worked example
Checklist
- List every AI user-agent you have decided about
- Give each bot its own named user-agent block
- Allow public content and disallow private or transactional paths
- Confirm Googlebot and Bingbot remain fully allowed
- Validate the file and publish it at the root
- Check logs to confirm bots obey the rules
Common mistakes
- Assuming a wildcard disallow will stop a named AI crawler that only reads its own block
- Blocking retrieval crawlers and then wondering why the brand never appears in AI answers
- Editing robots.txt in a way that accidentally disallows classic search engines
30-minute experiment
KPIs to track
- Number of AI crawlers correctly honouring rules
- Frequency of brand citations in AI answers
- AI-crawler requests to disallowed paths
FAQs
Will blocking GPTBot remove me from ChatGPT answers?
Not necessarily. Training and answer-time retrieval can use different user-agents, so blocking a training crawler need not remove you from live cited answers, but you must check each bot separately.
Does robots.txt actually stop AI crawlers?
Well-behaved crawlers honour it, but it is a request, not a lock. For content you must protect, combine robots rules with authentication or firewall-level blocking.
Recommended next steps
Where this fits - and what's next
The SearchScore path from a problem you feel to visibility you can measure.