What it does
Reviews robots.txt against AI and search crawlers and recommends an access policy aligned to your strategy.
Use when deciding which AI crawlers to allow or block, or auditing whether your robots.txt is accidentally hiding you from AI answers.
What you need first
- current robots.txt contents
- AI visibility goal (want AI citations or protect content)
- site type
The prompt
You are reviewing crawler access using the SearchScore Method.
CURRENT robots.txt:
{robots_txt}
AI VISIBILITY GOAL: {goal}
SITE TYPE: {site_type}
Task:
1. Parse the current rules and list what each user-agent group is allowed and disallowed.
2. Identify the major crawlers relevant here (search: Googlebot, Bingbot; AI: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) and state whether each is currently allowed, blocked, or unspecified.
3. Flag mistakes: blocking crawlers you depend on, blocking CSS/JS, disallowing indexable sections, or contradictory rules.
4. Recommend an access policy aligned to the goal - note the trade-off that blocking AI training crawlers can also reduce AI-answer visibility.
5. Output a corrected robots.txt block with comments explaining each decision.
Return the audit, the policy recommendation, and the corrected file. British English. Note that robots.txt is advisory, not enforcement.