A text file at a site's root that tells crawlers which paths they may or may not request.
ID
SS-DF-019
Confidence
High · 92
Evidence
Strong
Updated
2026-07-08
Definition
A text file at a site's root that tells crawlers which paths they may or may not request.
Robots.txt is a plain-text file served from the domain root that uses the Robots Exclusion Protocol to allow or disallow crawler access to specific paths. It controls crawling, not indexing: a disallowed URL can still be indexed without its content if other pages link to it, so it should never be relied on to hide a page from search. It is best used to keep crawlers away from low-value or infinite URL spaces and to point them to the XML sitemap.