How AI retrieval works: what happens between your website and a ChatGPT answer

Web-enabled AI systems do not all use the same retrieval stack. Some rely on proprietary search indexes, some on provider-specific crawlers, some on Google Search, and many combine retrieval with model knowledge. This guide explains the common concepts without pretending every engine uses the same chunk sizes, ranking stages or citation rules.

SearchScore note: the audit can measure observable website readiness such as crawl access, structure and entity clarity. It cannot see a provider’s private retrieval index or prove that a page “failed at the retrieval stage”, so hidden-stage percentages should not be presented as measured facts.

How does the AI retrieval pipeline work?

A practical cross-provider model has four broad layers, but their implementation and order can vary:

1. Crawl - AI crawlers discover and download your pages

2. Index - Content is parsed, chunked and stored in a knowledge base

3. Retrieve - User queries are matched to the most relevant content chunks

4. Generate - The LLM selects which sources to cite in its answer

The model is useful for diagnosis, not for assigning a hidden failure reason. A site can be accessible yet absent because of relevance, indexing, ranking, source competition or answer-generation choices. Measure the target answers rather than asserting which private stage rejected the page.

Stage 1: crawl - how AI crawlers discover your pages

Provider-specific search crawlers include OAI-SearchBot, PerplexityBot and Claude-SearchBot. Google Search uses Googlebot. GPTBot and ClaudeBot are separate training crawlers. Providers may use links, sitemaps and other discovery systems, but their complete discovery architecture is not public.

Which crawlers matter?

The four most important AI search crawlers in 2026 are OAI-SearchBot (OpenAI/ChatGPT search), PerplexityBot (Perplexity), Claude-SearchBot (Anthropic/Claude) and Googlebot (Google, including the index AI Overviews cites from). Note these are distinct from the training crawlers GPTBot and ClaudeBot, which do not govern search visibility (ByteDance). Each has its own user agent string and each checks robots.txt independently. Blocking one does not block the others.

What blocks AI crawlers?

Robots.txt and edge/WAF rules can block a documented search crawler. Rendering can also affect what a retrieval system receives. SearchScore’s July 2026 interactive sample found 6.9% of sites blocked at least one major AI-related crawler or control checked by the audit, not more than 40%.

This is step zero. If crawlers cannot access your content, nothing else in the pipeline matters. Check your robots.txt configuration for AI crawlers before doing anything else.

Stage 2: index - how AI systems build their knowledge base

After discovery, providers may index, parse, segment or fetch material on demand. Perplexity publishes details of its own index and reranking stack; Google Search has its own index; other providers disclose less. Do not assume one universal “AI knowledge base”.

What matters at the indexing stage?

Observable readiness checks include ordinary indexability, accessible text, accurate structured data where appropriate and clear page purpose. llms.txt is an optional content-map convention and is not documented as a primary indexing reference by the major search providers.

Keyword density, which dominated traditional SEO thinking, has near-zero impact at this stage. The AI is not counting keywords. It is parsing meaning, identifying entities and extracting factual claims.

Why structured data matters more than keywords

Structured data can expose page and entity relationships in a standard form to systems that consume Schema.org. It does not independently verify the claims, and there is no controlled basis for turning “missing schema” into a lower universal citation probability. Use the markup that truthfully describes visible content.

Stage 3: retrieve - how RAG matches queries to content

This is the bottleneck. This is where most sites lose.

Retrieval systems can operate at document, passage or sub-document level. Perplexity, for example, publicly describes both document- and sub-document ranking. Other systems may use different segmentation strategies. There is no universal 100-300 word chunk rule.

What makes content retrievable?

Content is retrievable when it can be chunked into self-contained passages that directly answer a specific question. The best-retrieved content has these characteristics:

• Question-format headings that the retrieval system can match to user queries (“What is AI retrieval?” matches “how does AI retrieve information”)

• Direct answers at the top of sections that work as standalone passages when extracted from the rest of the page

• One idea per paragraph so chunks are coherent and self-contained

• Specific, factual language rather than vague or abstract statements

• Clear heading hierarchy (H2, H3) that defines the semantic structure of the page

Content that fails retrieval is typically long, meandering, preamble-heavy or structured around marketing messages rather than answers. If the first 200 words of a section are brand positioning (“At Acme Corp, we believe in empowering…”), the retrieval system has nothing useful to extract.

Heading structure and passage clarity

Descriptive headings and coherent paragraphs help people and machines understand page structure. They may also make passage-level retrieval easier in systems that segment documents, but headings are not guaranteed chunk boundaries and should be written for the reader first.

Stage 4: generate - how the lLM chooses which sources to cite

After retrieval, a model or answer system uses the returned context to produce a response. Providers do not publish a universal candidate count or a three-factor source-selection formula. Relevance, source quality, freshness for current queries and clear factual content are sensible dimensions to inspect, but their weights differ.

Authority signals

The LLM favours sources that demonstrate expertise and trustworthiness. Author bylines, cited data, professional presentation and consistent entity signals all contribute. A small site with strong authority signals can be cited over a household name with weak ones.

Recency

For topics where timeliness matters, current information can be important. Refresh pages when the facts change and use honest dates. Do not assume a newer timestamp automatically outranks older but still authoritative material.

Extraction clarity

The LLM favours sources it can accurately paraphrase. Content that is clear, specific and factual is easier to extract than content that is vague, hedged or buried in jargon. The clearer your content, the more confidently the LLM can cite it.

The bottleneck: why most sites fail at stage 3

SearchScore can identify readiness gaps such as inaccessible content, poor structure and inconsistent entity data, but it cannot observe a private provider’s hidden retrieval rejection. Use structure improvements where they improve the page, then confirm impact by rerunning the actual buyer questions.

5 things you can do today to improve your retrieval odds

1. Check AI crawler access (5 minutes)

Visit your robots.txt and confirm OAI-SearchBot, PerplexityBot and Claude-SearchBot are not blocked. This is table stakes. If crawlers cannot reach your content, nothing else works.

2. Check indexability and page accessibility

Confirm important public pages are linked, indexable where relevant, and return usable content to the provider or upstream search system. If you maintain llms.txt, treat it as optional housekeeping rather than an immediate retrieval advantage.

3. Add question-format headings to your top 5 pages (1 hour)

Restructure your most important page headings as questions. “What is [X]?” “How does [Y] work?” “Why does [Z] matter?” These match the natural-language queries users type into AI search.

4. Front-load direct answers in every section (1 hour)

Open each section with a one-to-two-sentence direct answer, then expand with supporting detail. This creates extractable chunks that the retrieval system can match to queries.

5. Add genuine FAQs only where they help

If users genuinely ask repeated questions, answer them clearly on the page. FAQPage can describe visible Q&A semantically where appropriate, but FAQ sections are not a universal “most retrievable” format and the markup does not amplify a proven AI citation signal.