◆Domain Domain Stable

Technical SEO

Making sure engines and AI crawlers can reach, render, understand and trust your pages - the plumbing beneath every ranking.

ID
SS-DM-002
Version
1.0
Confidence
High · 88
Evidence
Strong
Updated
2026-07-08
Review
2026-10-08

Purpose

Overview

Technical SEO is the discipline of removing every mechanical barrier between your content and the systems that rank and cite it: crawlability, indexability, rendering, site architecture, structured data, speed and the newer question of whether AI crawlers (GPTBot, PerplexityBot, Google-Extended) are even allowed in. In an AI-search world, technical health is not hygiene - it is the difference between being a citable source and being invisible.

Business problem

Great content earns nothing when pages can't be crawled, rendered, indexed or parsed by AI retrievers.

KPIs for this domain

  • Indexation rate
  • Crawl-error count
  • AI crawler access coverage
  • Core Web Vitals pass rate

Inside this domain

Capabilities 5

◈Capability 88 Technical Health Optimisation Remove the crawl, render, index and structured-data barriers that stop engines and AI retrievers from using your pages. ◈Capability 81 Internal Link Architecture Structure internal links so authority flows to money pages, topics cluster clearly, and no important page is orphaned. ◈Capability 85 Indexability Management Ensure the right pages are indexable and the wrong ones are not, so search engines can find and rank valuable content. ◈Capability 79 Crawl Optimisation Direct crawl budget to important pages by removing waste, fixing structure and improving discovery. ◈Capability 81 Schema Optimisation Select and implement the right structured data so pages qualify for rich results and are understood by AI engines.

Frameworks 6

▤Framework 81 Internal Link Architecture Framework Design internal links as deliberate structure that routes authority to money pages and makes topics legible to engines. ▤Framework 86 Technical Audit Framework A structured sweep of crawl, index, render, structured-data and speed health, prioritised by visibility impact. ▤Framework 84 Indexability Framework Ensure the right pages are indexed and the wrong ones are not, so crawl budget and authority are not wasted. ▤Framework 82 Crawlability Framework Make sure every crawler that matters - including AI bots - can discover and fetch your important pages efficiently. ▤Framework 80 Schema Selection Framework Choose and implement the structured data that actually helps engines and AI understand and trust a page. ▤Framework 78 Log File Analysis Framework A repeatable method for reading server logs to understand how search and AI crawlers actually spend budget on a site.

Patterns 9

◇Pattern 85 Pages Not Indexed Important pages are missing from the index. ◇Pattern 81 Poor Internal Linking Internal links fail to route authority or express topic structure. ◇Pattern 80 Schema Errors Structured data is missing, invalid or misdescribes the page. ◇Pattern 82 Duplicate Content Near-identical pages compete and dilute signals. ◇Pattern 82 Orphan Pages Pages exist but nothing links to them internally. ◇Pattern 84 Broken Internal Links Internal links point to missing or redirected pages. ◇Pattern 82 Slow Site Pages load slowly and fail Core Web Vitals. ◇Pattern 79 Crawl Budget Waste Crawlers spend budget on low-value URLs instead of important pages. ◇Pattern 81 Redirect Issues Redirect chains, loops or wrong-type redirects harm crawling and equity.

Decisions 10

⌥Decision 83 Should I improve internal linking? When internal linking is the highest-leverage fix available. ⌥Decision 80 Should I add schema? When structured data will genuinely help engines and AI understand a page. ⌥Decision 85 Should I fix indexing first? When technical indexing issues must be resolved before any content work. ⌥Decision 80 Should I improve page speed? When speed work is worth prioritising over other fixes. ⌥Decision 82 Should I noindex this section? When a section of your site adds no search value but consumes crawl budget or dilutes quality signals. ⌥Decision 79 Should I migrate my URL structure? When your current URL scheme is messy or misaligned and you are weighing a site-wide migration. ⌥Decision 82 Should I fix Core Web Vitals now? When your page-experience metrics are poor and you must decide whether to prioritise them over content work. ⌥Decision 81 Should I redirect or keep this page? When a page overlaps another and you must decide whether to redirect it or keep it live. ⌥Decision 83 Should I add schema to this template? When a page template could carry structured data and you are weighing the build effort against the payoff. ⌥Decision 82 Should I canonicalise these duplicate pages? When near-duplicate URLs exist and you are choosing between canonical tags, redirects or leaving them.

Guides 13

▶Guide 85 How to run a technical SEO audit that finds problems worth fixing A repeatable method for auditing crawlability, indexation and rendering, then ranking the findings by business impact. ▶Guide 85 How to fix pages that aren't being indexed A diagnostic path for finding why valuable pages are excluded from the index and getting them back in. ▶Guide 82 How to optimise crawl budget on a large site Practical steps to stop crawlers wasting effort on low-value URLs so your important pages get crawled and refreshed faster. ▶Guide 80 How to configure robots.txt for AI crawlers How to decide and control whether GPTBot, PerplexityBot, Google-Extended and other AI agents can access your site. ▶Guide 84 How to implement structured data that helps AI understand your pages A selection-and-implementation method for schema.org markup that earns rich results and clarifies entities for AI systems. ▶Guide 84 How to design a crawlable site architecture How to structure URLs, navigation and internal links so crawlers reach every important page and understand how they relate. ▶Guide 84 How to pass Core Web Vitals A focused method for diagnosing and fixing the loading, interactivity and stability metrics that affect user experience and ranking. ▶Guide 82 How to make a JavaScript site render for search and AI How to ensure content built with client-side JavaScript is actually seen, indexed and cited by crawlers. ▶Guide 84 How to fix redirect chains and status codes How to audit and clean up redirects and HTTP responses so crawlers and users reach the right page in one hop. ▶Guide 82 How to run a log-file analysis How to use server logs to see exactly how search and AI crawlers behave on your site, and act on it. ▶Guide 83 Advanced Internal Link Architecture An expert framework for engineering internal links so authority flows to the pages that matter and no valuable page is left orphaned or buried. ▶Guide 85 Advanced Indexability Management A rigorous method for controlling exactly which pages enter the index, why the rest are excluded, and how to fix the silent leaks that keep valuable pages out. ▶Guide 82 Advanced Crawl Optimisation An expert approach to conserving crawl budget so search engines spend their limited attention on the pages that earn revenue, using log files as ground truth.

Definitions 36

§Definition 88 Crawl Budget The number of URLs a search engine is willing and able to crawl on a site within a given period. §Definition 92 Robots.txt A text file at a site's root that tells crawlers which paths they may or may not request. §Definition 91 XML Sitemap A structured file listing a site's important URLs to help search engines discover and prioritise them. §Definition 90 Canonical Tag An HTML link element that tells search engines which URL is the preferred version of duplicate or similar pages. §Definition 91 Noindex A directive that instructs search engines not to include a page in their index. §Definition 87 Nofollow A link attribute that tells search engines not to pass ranking credit through a link. §Definition 86 Hreflang An annotation that tells search engines which language and regional version of a page to show each user. §Definition 85 Pagination The practice of splitting a large set of content across a sequence of numbered pages. §Definition 91 301 Redirect A permanent server instruction that sends users and crawlers from an old URL to a new one. §Definition 88 Redirect Chain A sequence of two or more redirects between the requested URL and the final destination. §Definition 90 HTTP Status Codes Numeric server responses that tell clients and crawlers the outcome of a request. §Definition 88 Rendering The process of executing a page's code to produce the final content a crawler or browser sees. §Definition 86 JavaScript SEO The discipline of making JavaScript-dependent content crawlable, renderable and indexable. §Definition 87 Server-Side Rendering Generating a page's full HTML on the server so it arrives ready to display and index. §Definition 82 Dynamic Rendering Serving a prerendered HTML version to crawlers while sending the JavaScript app to users. §Definition 89 Core Web Vitals A set of Google metrics measuring the loading, interactivity and visual stability of a page. §Definition 88 Largest Contentful Paint A metric measuring how long it takes for the largest visible element to render on screen. §Definition 85 Interaction to Next Paint A metric measuring how quickly a page visually responds to user interactions. §Definition 88 Cumulative Layout Shift A metric measuring how much a page's visible content unexpectedly moves during loading. §Definition 86 Time to First Byte The time between a request and the first byte of the response arriving from the server. §Definition 89 Mobile-First Indexing Google's practice of using the mobile version of a page as the primary basis for indexing and ranking. §Definition 89 Structured Data Standardised markup that describes a page's content to search engines in a machine-readable form. §Definition 88 Schema.org A shared vocabulary of types and properties used to mark up structured data on the web. §Definition 88 JSON-LD The recommended script-based format for embedding structured data in a page. §Definition 86 Rich Results Enhanced search listings that display extra visual features drawn from structured data. §Definition 85 Breadcrumb Markup Structured data that describes a page's position within the site hierarchy for search engines. §Definition 85 Log File Analysis Examining server access logs to see exactly how search engine crawlers interact with a site. §Definition 86 Crawl Depth The number of clicks from the homepage required to reach a given page. §Definition 85 Index Bloat The condition of having many low-value or duplicate URLs indexed, diluting a site's quality signals. §Definition 84 Faceted Navigation Filter and sort controls that generate many URL combinations from a set of listings. §Definition 85 URL Parameters Query-string values appended to a URL that modify or track the content it returns. §Definition 87 Site Architecture The way a website's pages are organised and interlinked into a coherent hierarchy. §Definition 87 Soft 404 A page that shows a not-found or empty state to users but returns a success status code. §Definition 85 Duplicate Content Substantially identical content that appears at more than one URL, on the same site or across different sites. §Definition 84 Anchor Text The visible, clickable words of a hyperlink, which give search engines context about the linked page. §Definition 84 Alt Text The descriptive text attribute on an image that conveys its meaning to screen readers and search engines.

Recommended next steps

    See the wider capability Crawl Optimisation Capability → Apply the method Crawlability Framework Framework → Follow the step-by-step Advanced Crawl Optimisation Guide →