The process by which search engines discover pages by following links and requesting URLs.
ID
SS-DF-016
Confidence
High · 93
Evidence
Strong
Updated
2026-07-08
Definition
The process by which search engines discover pages by following links and requesting URLs.
Crawling is how automated bots such as Googlebot find and fetch web pages, moving from link to link and from sitemaps to discover new and updated URLs. It is the first stage of the search pipeline: a page cannot be indexed or ranked, or cited by an AI answer engine, until a crawler has successfully reached and downloaded it. Technical barriers such as blocked resources, server errors or poor internal linking can prevent effective crawling.