Definition
A catch-all label for automated AI-related fetchers, but crawler roles differ: search, training and user-triggered agents must be treated separately.
'AI crawler' is a convenient umbrella term, not one technical role. Providers use separate agents for different jobs: search/discovery crawlers such as OAI-SearchBot and PerplexityBot, training crawlers such as GPTBot, control tokens such as Google-Extended, and user-triggered fetchers. Robots.txt decisions should therefore be made per documented role rather than by assuming that blocking or allowing every AI-related token has the same visibility consequence.