Executive summary
llms.txt is an emerging, proposed convention: a Markdown file at your root that lists your key pages with short descriptions so AI systems can find and understand them efficiently. Adoption by major engines is not guaranteed, but the file is cheap to publish, harmless, and a useful discipline for curating your most citable content. Treat it as a low-cost bet, not a ranking guarantee.
What this helps you decide
Whether to publish an llms.txt file and which pages it should highlight.
Business problem
AI agents crawling messy sites waste effort on navigation, boilerplate and low-value pages, and may miss the content you most want cited. Without a curated guide you have no way to point them at your best material.
Step-by-step process
-
1
Decide if it is worth it
The format is a proposal not yet universally consumed by major engines. Publish it if the curation effort is low and you value a clean content map, but do not expect it alone to change citations.
-
2
Create the file at your root
Place a Markdown file at /llms.txt. Start with an H1 of your brand name and a short blockquote summarising what your organisation does.
-
3
Curate your best pages
Under themed H2 sections, list links to your most important, citable pages as Markdown bullets, each with a one-line description. Include only genuinely valuable content, not your whole sitemap.
-
4
Write model-friendly descriptions
Describe each link in plain, factual language a model can use to judge relevance. Avoid marketing fluff and keep each description to a single clear sentence.
-
5
Consider clean content endpoints
Where practical, offer clean Markdown versions of key pages, referenced from the file, so agents can consume the substance without navigation and boilerplate.
-
6
Keep it current
Review the file whenever you publish or retire cornerstone content so it never points at stale or removed pages.
-
7
Validate and monitor
Confirm the file loads as plain text, links resolve, and watch logs for AI agents fetching it to gauge whether it is being used.
Worked example
Checklist
- File is served at /llms.txt as plain text Markdown
- H1 brand name and a concise summary blockquote present
- Only genuinely citable pages are listed
- Each link has a single plain-language description
- Links resolve and are reviewed on content changes
Common mistakes
- Dumping the entire sitemap instead of curating cornerstone pages
- Treating llms.txt as a guaranteed ranking or citation lever
- Publishing once and letting the links go stale
30-minute experiment
KPIs to track
- AI-agent fetches of /llms.txt
- Share of listed pages later cited by AI engines
FAQs
Do major AI engines actually read llms.txt?
Support is partial and evolving. Some tools consume it, but the big engines have not committed to it universally, so treat it as a low-cost, forward-looking measure.
Is llms.txt the same as robots.txt?
No. robots.txt controls crawler access, while llms.txt curates and describes your best content for AI systems. They are complementary, not substitutes.
Recommended next steps
Where this fits - and what's next
The SearchScore path from a problem you feel to visibility you can measure.