Pillar Article
Technical GEO: How to optimise your website for AI search
The technical changes that make the biggest difference to your AI search visibility. From robots.txt and llms.txt to structured data and schema markup - a complete technical GEO guide.
Updated
Video transcript
Want AI to recommend your business? Start with these three fixes. They move the needle more than anything else you can do. First, open the door. Robots.txt lets AI in, llms.txt tells it what matters. Second, speak its language. Schema turns your pages into quotable facts. Third, keep it clean and fast. If AI can't parse it, it can't cite it. Save this before your next website update, free on SearchScore.
What is technical GEO?
Technical GEO is the work of configuring your website so AI engines can reach it, read it and quote it. In practice it is four things: letting AI crawlers in via robots.txt, publishing an llms.txt file, adding schema markup, and serving your content as clean HTML that does not need JavaScript to appear.
It is deliberately unglamorous, and that is the point. Most GEO advice is about writing better content. Technical GEO is about making sure the content you already have is machine-readable at all, and it is where the fastest wins live: two of the four fixes take under an hour, and neither requires a developer.
The order below is the order to do them in. Everything downstream depends on the first one, because a blocked crawler cannot be persuaded by better schema.
Key Takeaway
Technical GEO requires configuring robots.txt and llms.txt files, along with structured data schemas and optimised content structure, to ensure AI crawlers can accurately parse and cite your website.
In this guide
- Prioritised: what to fix first
- AI crawler permissions in robots.txt
- Schema markup for AI citation
- Structured data implementation
- Platform and performance signals
How to do technical GEO, in order
Not all technical GEO changes are equal. SearchScore’s analysis of 1,000,000+ website audits found that three issues account for the majority of AI search invisibility, and two of them take under an hour to fix. Data: SearchScore audit corpus, July 2026.
| # | Fix | Impact | Time | Who does it |
|---|---|---|---|---|
| 1 | Unblock AI crawlers in robots.txt | Highest | 10 minutes | Anyone with file access |
| 2 | Create an llms.txt file | Highest | 30 minutes | Anyone who can write |
| 3 | Add schema markup | High | 1 to 4 hours | Developer, or a CMS plugin |
| 4 | Fix page structure and rendering | Medium | Ongoing | Developer |
Steps 1 and 2 are binary: either the file is right or it is not, and getting them wrong makes everything after them irrelevant. Steps 3 and 4 are incremental, so partial progress still counts.
If you only have an afternoon, do 1 and 2 and stop. It is a better use of the time than half-implementing schema on a site that crawlers cannot reach.
AI crawler permissions in robots.txt
Your robots.txt file tells web crawlers which parts of your site they can access. The problem is that most robots.txt files were written before AI search engines existed - and many contain blanket rules that accidentally block AI crawlers alongside spam bots.
The major AI crawlers and their user-agent names:
- GPTBot - OpenAI / ChatGPT
- PerplexityBot - Perplexity AI
- ClaudeBot - Anthropic / Claude
- Googlebot - Google AI Overviews (uses standard Googlebot)
- anthropic-ai - Anthropic web crawler
- cohere-ai - Cohere
Common mistake: Using User-agent: * with Disallow: / to block all bots will also block every AI crawler. This is the single most damaging GEO error: across 6,944 websites audited interactively by SearchScore (July 2026), 6.9% block at least one major AI crawler, most without realising it.
To allow all major AI crawlers, add these lines to your robots.txt:
# Allow major AI crawlers
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: cohere-ai
Allow: /
If you need to block AI training data collection while allowing search, use more specific directives. OpenAI, Anthropic and others honour different bot names for training versus live retrieval.
Creating your llms.txt file
llms.txt is a plain text file placed at the root of your website (e.g. yoursite.com/llms.txt) that gives AI language models structured guidance about your site. Think of it as a sitemap for AI - not just where pages are, but what your site is, what your most important content covers, and how an AI should understand your brand.
The format is simple Markdown. A basic llms.txt looks like this:
# YourBrand
> One-line description of what your website/business does.
## About
[Brief description of who you are, what you do, and who you serve]
## Key pages
- [Home](https://yoursite.com/): Main landing page
- [About](https://yoursite.com/about/): Company background and team
- [Blog](https://yoursite.com/blog/): Articles and guides
## Key topics
This site covers [your main topic areas].
Our content is written by [credentials].
## Contact
[[email protected]]
Beyond the basics, you can also include a detailed llms-full.txt that contains the complete text of your most important pages - making it trivial for AI models to ingest your content without crawling your full site.
Quick win: In SearchScore’s interactive audit data (6,944 websites, July 2026), roughly 7 in 10 websites (69.7%) have no llms.txt file. Simply creating one puts you ahead of almost all of your competitors from an AI search perspective.
Schema markup for AI citation
Schema.org markup is structured data embedded in your HTML that tells machines what your content means. Google has required it for rich results for years - but for GEO, it is even more important. AI engines use schema to verify facts, understand entities, attribute authorship and decide whether to cite your content.
Organisation schema
Organisation schema establishes your brand as a known entity. It should include your official name, URL, logo, contact details, social media profiles and, where applicable, your Wikipedia or Wikidata URL. This is the foundation of brand authority for AI citation.
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Your Company Name",
"url": "https://yoursite.com",
"logo": "https://yoursite.com/logo.png",
"sameAs": [
"https://twitter.com/yourhandle",
"https://linkedin.com/company/yourcompany",
"https://en.wikipedia.org/wiki/YourCompany"
]
}
Article schema
Every blog post and article should have Article schema with a named author, a datePublished, and a publisher reference. This is how AI engines attribute content to real, verified people - a critical EEAT signal.
FAQPage schema
FAQPage schema is one of the most powerful GEO signals available. AI engines that synthesise answers frequently pull from structured Q&A content - and FAQ schema makes your Q&A pairs directly machine-readable. Add it to any page with questions and answers.
Structured data: the bigger picture
Beyond the core three schema types, consider adding structured data relevant to your business type:
- LocalBusiness - for businesses with a physical location
- Product - for ecommerce and software products
- Person - for author pages and personal brands
- HowTo - for instructional content (AI engines love step-by-step guides)
- BreadcrumbList - helps AI understand your site hierarchy
- WebSite with SearchAction - signals your site as a navigable entity
Implement schema as JSON-LD in the <head> of your pages. It is easier to maintain than inline microdata and is the format preferred by both Google and AI crawlers.
Platform and performance signals
AI crawlers face the same technical barriers as other bots. Slow load times, JavaScript-heavy rendering, broken pagination and inconsistent canonical URLs all reduce how effectively AI engines can parse your content.
- Core Web Vitals - fast LCP and low CLS improve crawl efficiency
- Semantic HTML - use proper heading hierarchy (H1 > H2 > H3), not divs styled to look like headings
- Alt text - all images labelled, helping AI understand visual content context
- Canonical tags - prevent AI engines from indexing duplicate content versions
- XML sitemap - ensure all important pages are discoverable
- HTTPS - a basic trust signal for all search engines, including AI
Technical GEO checklist
- ☐ GPTBot, ClaudeBot and PerplexityBot are not blocked in robots.txt
- ☐ llms.txt file exists at domain root
- ☐ llms.txt includes accurate site description, key pages and key topics
- ☐ Organisation schema implemented on homepage
- ☐ Article schema on all blog posts with named author
- ☐ FAQPage schema on key pages
- ☐ Person schema on author bio pages
- ☐ All schema validated with Google Rich Results Test
- ☐ Canonical tags on all pages
- ☐ XML sitemap submitted to Google Search Console
- ☐ HTTPS active across entire site
- ☐ No JavaScript rendering required to access main content
- ☐ H1 > H2 > H3 heading hierarchy consistent on all pages
- ☐ Image alt text complete
How to check whether it worked
Technical GEO has an unusually clean feedback loop, because most of it is verifiable the same day. Work through these in order.
1. Confirm the files are live and correct. Load yoursite.com/robots.txt and yoursite.com/llms.txt in a browser. They must return HTTP 200 and plain text, not a 404 page styled to look like content. A surprising number of sites publish an llms.txt that their CDN or framework silently rewrites into HTML.
2. Confirm your content exists without JavaScript. View source, or run curl -s yoursite.com | grep "a distinctive sentence from your page". If the sentence is not in the raw HTML, most AI crawlers will not see it either. This is the single most common cause of a technically tidy site being uncitable.
3. Validate the schema. Run your key pages through Google’s Rich Results Test and the Schema.org validator. Invalid JSON-LD is usually ignored entirely rather than partially credited, so a single syntax error can waste the whole implementation.
4. Check crawler access in your logs. Look for GPTBot, ClaudeBot, PerplexityBot and Googlebot in your server logs or CDN analytics. Seeing them arrive is the only direct proof that step 1 worked. Expect days rather than hours, and note that crawl frequency is low for small sites.
5. Re-score the site. A free SearchScore audit checks these signals across 250+ tests and returns a GEO score in about 60 seconds, with no account. Run it before you start and again a week later so you have a before-and-after rather than an opinion.
A caveat worth stating plainly: none of the above proves you are being cited. It proves you are eligible to be. Readiness and citation are different measurements, and the gap between them is usually filled by content quality and third-party corroboration rather than more technical work. Checking whether engines actually name you means putting real questions to them, which is what SearchScore Tracker does across six engines.
Frequently asked questions
How long does technical GEO take to show results?
The fixes themselves take hours. The effect takes weeks. AI engines do not re-crawl on demand, and several build their answers from index snapshots refreshed on their own schedule. Expect to see crawler activity within days and any change in citation over four to eight weeks.
Is llms.txt actually used by AI engines?
Honestly, adoption is still uneven. No major engine has committed publicly to reading it as a ranking input. It is worth doing anyway because it costs thirty minutes, it cannot hurt, and it forces you to write a clear machine-readable summary of what your site is, which is useful regardless. Treat it as cheap insurance, not a guaranteed lever. See llms.txt examples for what a good one looks like.
Do I need schema markup if my site already ranks well on Google?
Yes, and this is the mistake most established sites make. Ranking well means Google understands your pages. Schema helps AI engines understand what your content means well enough to quote it and attribute it correctly. A page can rank first on Google and still be skipped by an engine assembling an answer, because ranking and citation use different selection criteria.
Will blocking AI crawlers protect my content?
It will stop you being cited. That is the whole trade. If your business depends on being recommended, blocking crawlers removes you from consideration entirely. If you are a publisher with a licensing strategy, blocking may be the right commercial call. It is a business decision, not a technical one, and it should be made deliberately rather than inherited from a robots.txt written years ago.
What is the single most common technical GEO failure?
Content that only exists after JavaScript runs. It is far more common than an outright crawler block, and much harder to spot, because the page looks perfect in a browser. Check the raw HTML, not the rendered page.
Does technical GEO differ from technical SEO?
It overlaps heavily and then diverges. Crawlability, clean HTML, canonical tags and site speed serve both. The GEO-specific additions are AI crawler permissions, llms.txt, and schema chosen for machine comprehension rather than rich snippets. If your technical SEO is sound you are most of the way there, and the remaining work is a short list.
Check your AI visibility
Free audit. Instant results. No sign-up required.
Check my score →
Continue reading: Technical GEO cluster
- How to create an llms.txt file (step by step) →
- AI bot permissions in robots.txt - complete guide →
- Schema markup for AI citation →
- Structured data types AI engines rely on →
- GEO technical audit checklist →
- llms.txt examples: what good files look like →
Explore All Categories
Sources & Further Reading
- OpenAI – GPTBot and OpenAI crawler documentation
- llmstxt.org – The llms.txt standard specification
- Schema.org – Getting started with structured data
- Academic research – GEO: Generative Engine Optimization (Aggarwal et al., arXiv)
Check your AI visibility
Enter your URL at SearchScore for a free AI visibility Score. See how ChatGPT, Perplexity and Google AI see your site - and exactly what to fix.
Track your AI citations weekly
SearchScore Tracker runs weekly (or daily) scans across ChatGPT, Claude, Gemini, Perplexity, Grok and DeepSeek. Get your baseline, free.
Related guides
Technical Geo AI bot permissions in robots.txt: The complete guide Technical Geo How to create an llms.txt file: Step-by-step guide Technical Geo GEO technical audit checklist: 25 items to check Technical Geo schema markup for AI citation: Which types matter most