An automated bot that fetches web content to train AI models or to ground and cite live AI answers.
ID
SS-DF-110
Confidence
Established · 78
Evidence
Established
Updated
2026-07-08
Definition
An automated bot that fetches web content to train AI models or to ground and cite live AI answers.
AI crawlers - such as GPTBot, PerplexityBot, ClaudeBot and Google-Extended - visit pages to collect content for model training or real-time retrieval. Whether you permit them, via robots.txt and server rules, determines if your content can enter an engine's knowledge and be cited in its answers. Managing AI crawler access is now a distinct technical discipline: too restrictive and you vanish from AI results, too open and you cede control over how your content is used.