Pillar Article
Technical GEO: How to optimise your website for AI search
The technical foundations that make a site eligible to be found and used by search and answer engines: crawl access, indexable HTML, clean architecture and accurate structured data. It also explains where optional conventions such as llms.txt fit - without treating them as ranking shortcuts.
Updated
Video transcript
Want AI to recommend your business? Start with these three fixes. They move the needle more than anything else you can do. First, open the door. Robots.txt lets AI in, llms.txt tells it what matters. Second, speak its language. Schema turns your pages into quotable facts. Third, keep it clean and fast. If AI can't parse it, it can't cite it. Save this before your next website update, free on SearchScore.
What is technical GEO?
Technical GEO is the work of making important pages reachable, indexable and unambiguous to the systems that may retrieve them. The foundations are ordinary technical SEO plus provider-specific search crawler access where relevant, clean HTML and accurate structured data. An llms.txt file is optional agent-readiness housekeeping, not a prerequisite for AI search.
It is deliberately unglamorous, and that is the point. Most GEO advice is about writing better content. Technical GEO is about making sure the content you already have is machine-readable at all, and it is where the fastest wins live: two of the four fixes take under an hour, and neither requires a developer.
The order below is the order to do them in. Everything downstream depends on the first one, because a blocked crawler cannot be persuaded by better schema.
Key Takeaway
Technical GEO requires deliberate crawler access, indexable content, clean technical architecture and structured data that truthfully describes the page. None of those guarantees a citation; they remove avoidable barriers so the page can be discovered and interpreted.
In this guide
- Prioritised: what to fix first
- AI crawler permissions in robots.txt
- Schema markup for AI citation
- Structured data implementation
- Platform and performance signals
How to do technical GEO, in order
Not all technical GEO changes are equal. SearchScore’s analysis of 1,000,000+ website audits found that three issues account for the majority of AI search invisibility, and two of them take under an hour to fix. Data: SearchScore audit corpus, July 2026.
| # | Fix | Impact | Time | Who does it |
|---|---|---|---|---|
| 1 | Unblock AI crawlers in robots.txt | Highest | 10 minutes | Anyone with file access |
| 2 | Create an llms.txt file | Optional | 30 minutes | Anyone who can write |
| 3 | Add schema markup | High | 1 to 4 hours | Developer, or a CMS plugin |
| 4 | Fix page structure and rendering | Medium | Ongoing | Developer |
Steps 1 and 2 are binary: either the file is right or it is not, and getting them wrong makes everything after them irrelevant. Steps 3 and 4 are incremental, so partial progress still counts.
If you only have an afternoon, do 1 and 2 and stop. It is a better use of the time than half-implementing schema on a site that crawlers cannot reach.
AI crawler permissions in robots.txt
Your robots.txt file tells web crawlers which parts of your site they can access. The problem is that most robots.txt files were written before AI search engines existed - and many contain blanket rules that accidentally block AI crawlers alongside spam bots.
The major AI crawlers and their user-agent names:
- OAI-SearchBot - OpenAI search discovery - GPTBot - OpenAI model training; decide separately from search access
- PerplexityBot - Perplexity AI
- ClaudeBot - Anthropic / Claude
- Googlebot - Google AI Overviews (uses standard Googlebot)
- Claude-SearchBot - Anthropic search indexing (governs Claude citation)
- ClaudeBot - Anthropic model training
- cohere-ai - Cohere
Common mistake: Using User-agent: * with Disallow: / to block all bots will also block every AI crawler. This is the single most damaging GEO error: across 6,944 websites audited interactively by SearchScore (July 2026), 6.9% block at least one major AI crawler, most without realising it.
To allow all major AI crawlers, add these lines to your robots.txt:
# Allow major AI crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: cohere-ai
Allow: /
If you need to block AI training data collection while allowing search, use more specific directives. OpenAI, Anthropic and others honour different bot names for training versus live retrieval.
Creating your llms.txt file
llms.txt is a proposed Markdown file at the root of a website (for example yoursite.com/llms.txt) that can provide a curated map of important content to tools that deliberately support the convention. Google Search says it does not use llms.txt and no major answer engine has documented it as a universal citation input, so treat it as optional agent-readiness housekeeping.
The format is simple Markdown. A basic llms.txt looks like this:
# YourBrand
> One-line description of what your website/business does.
## About
[Brief description of who you are, what you do, and who you serve]
## Key pages
- [Home](https://yoursite.com/): Main landing page
- [About](https://yoursite.com/about/): Company background and team
- [Blog](https://yoursite.com/blog/): Articles and guides
## Key topics
This site covers [your main topic areas].
Our content is written by [credentials].
## Contact
[[email protected]]
Beyond the basics, you can also include a detailed llms-full.txt that contains the complete text of your most important pages - making it trivial for AI models to ingest your content without crawling your full site.
Context: SearchScore’s July 2026 interactive audit sample found many sites had no valid llms.txt. That makes the file uncommon, not proven to be a competitive ranking advantage. Publish one when the maintenance cost is low or a tool you care about supports it; prioritise crawlability, indexability, content quality and authority first.
Schema markup for AI citation
Schema.org markup is structured data embedded in your HTML that tells machines what your content means. Google has required it for rich results for years - but for GEO, it is even more important. AI engines use schema to verify facts, understand entities, attribute authorship and decide whether to cite your content.
Organisation schema
Organisation schema establishes your brand as a known entity. It should include your official name, URL, logo, contact details, social media profiles and, where applicable, your Wikipedia or Wikidata URL. This is the foundation of brand authority for AI citation.
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Your Company Name",
"url": "https://yoursite.com",
"logo": "https://yoursite.com/logo.png",
"sameAs": [
"https://twitter.com/yourhandle",
"https://linkedin.com/company/yourcompany",
"https://en.wikipedia.org/wiki/YourCompany"
]
}
Article schema
Every blog post and article should have Article schema with a named author, a datePublished, and a publisher reference. This is how AI engines attribute content to real, verified people - a critical EEAT signal.
FAQPage schema
FAQPage can describe genuine question-and-answer content in a machine-readable Schema.org format. Use it only when the FAQs are visibly present. Google retired FAQ rich results in May 2026, and there is no controlled evidence that FAQPage markup itself increases AI citation frequency.
Structured data: the bigger picture
Beyond the core three schema types, consider adding structured data relevant to your business type:
- LocalBusiness - for businesses with a physical location
- Product - for ecommerce and software products
- Person - for author pages and personal brands
- HowTo - for instructional content (AI engines love step-by-step guides)
- BreadcrumbList - helps AI understand your site hierarchy
- WebSite with SearchAction - signals your site as a navigable entity
Implement schema as JSON-LD in the <head> of your pages. It is easier to maintain than inline microdata and is the format preferred by both Google and AI crawlers.
Platform and performance signals
AI crawlers face the same technical barriers as other bots. Slow load times, JavaScript-heavy rendering, broken pagination and inconsistent canonical URLs all reduce how effectively AI engines can parse your content.
- Core Web Vitals - fast LCP and low CLS improve crawl efficiency
- Semantic HTML - use proper heading hierarchy (H1 > H2 > H3), not divs styled to look like headings
- Alt text - all images labelled, helping AI understand visual content context
- Canonical tags - prevent AI engines from indexing duplicate content versions
- XML sitemap - ensure all important pages are discoverable
- HTTPS - a basic trust signal for all search engines, including AI
Technical GEO checklist
- ☐ Search/retrieval crawlers you rely on (for example OAI-SearchBot, Claude-SearchBot and PerplexityBot) are not accidentally blocked
- ☐ If you maintain llms.txt, the file exists at domain root and is valid Markdown
- ☐ llms.txt includes accurate site description, key pages and key topics
- ☐ Organisation schema implemented on homepage
- ☐ Article schema on all blog posts with named author
- ☐ FAQPage schema on key pages
- ☐ Person schema on author bio pages
- ☐ Schema validates in Schema.org Validator; use Google Rich Results Test only for currently supported Google rich-result types
- ☐ Canonical tags on all pages
- ☐ XML sitemap submitted to Google Search Console
- ☐ HTTPS active across entire site
- ☐ No JavaScript rendering required to access main content
- ☐ H1 > H2 > H3 heading hierarchy consistent on all pages
- ☐ Image alt text complete
How to check whether it worked
Technical GEO has an unusually clean feedback loop, because most of it is verifiable the same day. Work through these in order.
1. Confirm the files are live and correct. Load yoursite.com/robots.txt and yoursite.com/llms.txt in a browser. They must return HTTP 200 and plain text, not a 404 page styled to look like content. A surprising number of sites publish an llms.txt that their CDN or framework silently rewrites into HTML.
2. Confirm your content exists without JavaScript. View source, or run curl -s yoursite.com | grep "a distinctive sentence from your page". If the sentence is not in the raw HTML, most AI crawlers will not see it either. This is the single most common cause of a technically tidy site being uncitable.
3. Validate the schema. Run the markup through Schema.org Validator. Use Google’s Rich Results Test only when the page targets a structured-data feature Google currently supports. Validation proves the markup is well formed; it does not prove a ranking or AI-citation effect.
4. Check crawler access in your logs. Look for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot in your server logs or CDN analytics. Seeing them arrive is the only direct proof that step 1 worked. Expect days rather than hours, and note that crawl frequency is low for small sites.
5. Re-score the site. A free SearchScore audit checks these signals and returns a GEO score in about 60 seconds, with no account. Run it before you start and again a week later so you have a before-and-after rather than an opinion.
A caveat worth stating plainly: none of the above proves you are being cited. It proves you are eligible to be. Readiness and citation are different measurements, and the gap between them is usually filled by content quality and third-party corroboration rather than more technical work. Checking whether engines actually name you means putting real questions to them, which is what SearchScore Tracker does across six engines.
Frequently asked questions
How long does technical GEO take to show results?
The fixes themselves take hours. The effect takes weeks. AI engines do not re-crawl on demand, and several build their answers from index snapshots refreshed on their own schedule. Expect to see crawler activity within days. Assistants typically pick up a change within one to eight weeks; some move in days.
Is llms.txt actually used by AI engines?
No major answer engine has documented llms.txt as a universal ranking or citation input, and Google explicitly says Search does not use it. It can still be worth maintaining as low-cost agent-readiness housekeeping or for a tool that explicitly supports the convention. Treat it as optional, not as a prerequisite. See llms.txt examples for the format.
Do I need schema markup if my site already ranks well on Google?
Yes, and this is the mistake most established sites make. Ranking well means Google understands your pages. Schema helps AI engines understand what your content means well enough to quote it and attribute it correctly. A page can rank first on Google and still be skipped by an engine assembling an answer, because ranking and citation use different selection criteria.
Will blocking AI crawlers protect my content?
It will stop you being cited. That is the whole trade. If your business depends on being recommended, blocking crawlers removes you from consideration entirely. If you are a publisher with a licensing strategy, blocking may be the right commercial call. It is a business decision, not a technical one, and it should be made deliberately rather than inherited from a robots.txt written years ago.
What is the single most common technical GEO failure?
Content that only exists after JavaScript runs. It is far more common than an outright crawler block, and much harder to spot, because the page looks perfect in a browser. Check the raw HTML, not the rendered page.
Does technical GEO differ from technical SEO?
It overlaps heavily and then adds provider-specific checks. Crawlability, indexability, clean HTML, canonicalisation, site speed and accurate structured data serve both. The extra work is mainly verifying the search/retrieval crawlers relevant to the engines you care about and measuring whether your pages actually appear in answers. llms.txt is optional, not a technical requirement.
Check your AI visibility
Free audit. Results in about 60 seconds. No sign-up required.
Continue reading: Technical GEO cluster
- How to create an llms.txt file (step by step) →
- AI bot permissions in robots.txt - complete guide →
- Schema markup for AI citation →
- Structured data types AI engines rely on →
- GEO technical audit checklist →
- llms.txt examples: what good files look like →
Explore All Categories
Sources & Further Reading
- OpenAI – OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User
- llmstxt.org – The llms.txt standard specification
- Schema.org – Getting started with structured data
- Academic research – GEO: Generative Engine Optimization (Aggarwal et al., arXiv)
Check your AI visibility
Enter your URL at SearchScore for a free AI visibility Score. See how ChatGPT, Perplexity and Google AI see your site - and exactly what to fix.
See whether your fixes worked
The Tracker re-checks your site every week and shows which scores moved. Start with the free scan, then see Tracker plans.
Related guides
Technical Geo AI bot permissions in robots.txt: The complete guide Technical Geo How to create an llms.txt file: Step-by-step guide Technical Geo GEO technical audit checklist: 25 items to check Technical Geo schema markup for AI citation: Which types matter most