Executive summary
AI crawlers fall into three jobs: training corpus collection, live retrieval for answers, and user-triggered fetches. To appear in AI answers you must allow the retrieval and user-fetch bots at minimum. robots.txt is the primary control, but honest bots vary and enforcement for misbehaving ones needs firewall rules. Decide per bot, verify in logs, and revisit as new agents appear.
What this helps you decide
Which AI crawlers to allow, block, or rate-limit, and how to enforce that.
Business problem
Many sites accidentally block the AI crawlers that drive citations, or block none and let bots hammer their servers. Both extremes cost you: one removes you from AI answers, the other wastes resources and exposes content you meant to protect.
Step-by-step process
-
1
Inventory the crawlers hitting you
Read server logs to list the AI user agents already requesting your site, such as GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended, and note their volume.
-
2
Separate training from retrieval intent
Distinguish bots that gather training data from those that fetch live content to answer a user. Retrieval and user-fetch bots are the ones that produce citations and clicks now.
-
3
Decide your policy per bot
Allow the retrieval and user-fetch agents you want citations from. Choose deliberately on training bots, weighing brand exposure against content-use concerns rather than blocking reflexively.
-
4
Encode it in robots.txt
Add explicit user-agent groups with allow or disallow rules. Keep the file readable and comment each decision so future edits do not undo your intent.
-
5
Enforce beyond robots.txt
Because robots.txt relies on good faith, use firewall or WAF rules and rate limits to stop bots that ignore it or overload your servers.
-
6
Protect sensitive areas
Ensure private, paywalled or thin sections are disallowed or authenticated so allowing AI crawlers on public content does not expose material you meant to keep back.
-
7
Verify and review
Confirm in logs that allowed bots now fetch successfully and blocked ones stop, and rereview the policy quarterly as new agents emerge.
Worked example
Checklist
- AI user agents hitting your site are inventoried from logs
- Retrieval and user-fetch bots you want citations from are allowed
- robots.txt decisions are explicit and commented
- Firewall or WAF enforces limits on bots that ignore robots.txt
- Sensitive sections remain disallowed or authenticated
Common mistakes
- Using a blanket disallow that silently removes you from AI answers
- Trusting robots.txt alone against bots that ignore it
- Blocking training bots reflexively without weighing brand exposure
30-minute experiment
KPIs to track
- Successful fetches by allowed AI retrieval bots
- Reduction in server load from rate-limited rogue bots
FAQs
Will allowing AI crawlers let them steal my content?
Allowing retrieval bots exposes only what is already public. Keep private or paywalled content disallowed or authenticated, and weigh training-bot access separately based on your content strategy.
Do all AI crawlers obey robots.txt?
The major declared ones generally do, but not all bots are honest. Back robots.txt with firewall and rate-limit rules to enforce your policy on those that ignore it.
Recommended next steps
Where this fits - and what's next
The SearchScore path from a problem you feel to visibility you can measure.