What it does
Reviews robots.txt against AI and search crawlers and recommends an access policy aligned to your strategy.
Use when deciding which search/discovery crawlers, training crawlers and control tokens to allow or block, or when auditing whether robots.txt is accidentally affecting search eligibility.
What you need first
- current robots.txt contents
- AI visibility goal (want AI citations or protect content)
- site type
The prompt
You are reviewing crawler access using the SearchScore Method.
CURRENT robots.txt:
{robots_txt}
AI VISIBILITY GOAL: {goal}
SITE TYPE: {site_type}
Task:
1. Parse the current rules and list what each user-agent group is allowed and disallowed.
2. Classify each relevant token by documented role before recommending anything: search/discovery crawler, training crawler, product-control token, user-triggered fetcher or ordinary search crawler. Treat OAI-SearchBot, GPTBot and Google-Extended as different roles, not one AI-bot class.
3. Flag mistakes: blocking a search crawler the site depends on, confusing a training opt-out with search eligibility, blocking CSS/JS, disallowing indexable sections, or contradictory rules.
4. Recommend an access policy aligned to the goal and explain the visibility/training trade-off for each named provider token.
5. Output a corrected robots.txt block with comments explaining each decision.
Return the audit, the role classification, the policy recommendation and the corrected file. British English. Note that robots.txt is advisory rather than access control for protected content.