Purpose
Provider crawler roles change and are easy to confuse. This SOP keeps search eligibility, training opt-outs and product-control tokens separate so visibility and content-use decisions remain deliberate.
When & who
Procedure
-
1
Inventory AI user agents
Inventory the current provider tokens and label each by role: ordinary search, search/discovery, training, product-control/grounding or user-triggered fetch.
-
2
Review robots.txt directives
Check which AI user agents are allowed or disallowed and confirm the rules match the intended visibility strategy.
-
3
Inspect edge and firewall rules
Verify the CDN, WAF or bot manager is not silently blocking or rate-limiting legitimate AI crawlers.
-
4
Confirm access with logs
Analyse server logs to confirm the intended AI crawlers are actually fetching key pages and returning 200 responses.
-
5
Assess llms.txt
Treat llms.txt as optional agent-readiness housekeeping. Maintain it only if the cost is low or a tool you use explicitly consumes it; do not treat it as search eligibility.
-
6
Reconcile with strategy
Where blocking is intentional, document the rationale, and where visibility is the goal, open access to the chosen agents.
-
7
Record and schedule review
Log the configuration decisions and set the next quarterly review to keep pace with new crawlers.
Tools
- Server log files
- Cloudflare
- Google Search Console
- SearchScore Tracker
Recommended next steps
Where this fits - and what's next
The SearchScore path from a problem you feel to visibility you can measure.