GPTBot vs OAI-SearchBot: which controls ChatGPT search visibility?

Short answer: Blocking GPTBot does not, by itself, take your website out of ChatGPT search. GPTBot is OpenAI’s training crawler. OAI-SearchBot is the crawler OpenAI uses for ChatGPT search. OpenAI documents them as separate settings (Overview of OpenAI Crawlers), so if you want to appear in ChatGPT search without allowing training, configure and check the two permissions separately.

The catch is the rest of your robots.txt. A catch-all rule, a firewall or a CDN can still keep OAI-SearchBot out even when you never meant to block it.

Which setting gives which result?

You want OAI-SearchBot GPTBot Notes
ChatGPT search, and allow training Allow Allow OpenAI may use one crawl for both purposes
ChatGPT search, but no training Allow Disallow The split OpenAI’s own documentation describes
Stay out of ChatGPT search answers Disallow Your choice You can still appear as a navigational link (see below)

How a crawler reads these rules depends on the whole file, not one line. The section on robots.txt groups explains why.

What does OpenAI say each crawler does?

OpenAI publishes one page listing its crawlers. All quotes below are from that page, checked on 12 November 2026.

User agent What OpenAI says it is for Controls ChatGPT search eligibility?
OAI-SearchBot “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features.” Yes
GPTBot “It is used to crawl content that may be used in training our generative AI foundation models.” No
ChatGPT-User Visits a page when users ask ChatGPT or a custom GPT a question. It “is not used for crawling the web in an automatic fashion.” No. OpenAI says it “is not used to determine whether content may appear in Search.”
OAI-AdsBot “OAI-AdsBot only visits pages submitted as ads.” No. Only relevant if you advertise in ChatGPT

OpenAI also says: “Each setting is independent of the others.”

Which crawler decides whether ChatGPT search can show me?

OAI-SearchBot. OpenAI’s wording is: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

So opting out does not guarantee your site never appears. It can still show up as a navigational link, for example when someone looks for your site by name. OpenAI does not define “navigational links” further on that page, so we would not read more into it than that.

OpenAI recommends allowing OAI-SearchBot in robots.txt if you want to appear in search results, and asks that you allow requests from its published IP ranges. That second point matters if a firewall or bot-protection rule sits in front of your site. Our evidence note on OpenAI’s publisher guidance on OAI-SearchBot summarises the same source.

Does blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search, on its own. OpenAI says “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Whether you allow it is a decision about training, separate from search eligibility. Our glossary entry for GPTBot gives the short definition.

Our own Fix-List card treats it the same way. The card for a blocked GPTBot reads “Decide whether to allow OpenAI’s training crawler (GPTBot) in robots.txt”. It is your call either way. Our audit score has not caught up with that card yet: it still counts a GPTBot block against the AI Citability category, so a deliberate block lowers your score even though it does not, on its own, take you out of ChatGPT search. We have a change to the scoring planned.

Allowing OAI-SearchBot makes your pages eligible for ChatGPT search. It does not mean ChatGPT will cite or recommend you for a given question. That depends on what is on the page and what else the engine finds.

How common is each block?

In SearchScore’s AI crawler policy study, measured on 18 August 2026 across 347,988 websites audited by SearchScore:

Read those with their limits. The 42.6% is a share of the GPTBot blockers, not of all websites. “Not blocked” includes sites where OAI-SearchBot is only reached by the catch-all group, so it does not mean the site named it. The sample is SearchScore’s audit corpus, not a random sample of the web, and robots.txt cannot tell us why a site was set up the way it was.

These figures were measured before we fixed a robots.txt parsing bug on 25 August 2026, one that could read OAI-SearchBot as blocked when a later group blocked GPTBot. They are kept as published. A separate re-measure on audits made after the fix, measured on 20 October 2026, uses a different sample and denominator (120,190 sites whose robots.txt has a rule reaching AI crawlers). In it, 10.39% blocked GPTBot, 5.48% blocked OAI-SearchBot, and 46.3% of GPTBot blockers left OAI-SearchBot allowed. The two studies are not a trend.

How do robots.txt groups decide which rule applies?

This is where most mistakes happen. A crawler does not read every line of the file. It picks one group.

The robots.txt standard, RFC 9309, says a crawler uses the group whose user-agent line matches its own name. “If no matching group exists, crawlers MUST obey the group with a user-agent line with the “*” value, if present.” Google describes the same behaviour for its crawlers: “Only one group is valid for a particular crawler”, and “User agent specific groups and global groups (*) are not combined” (Google Search Central).

Two practical consequences:

  1. If OAI-SearchBot has no group of its own, the User-agent: * group decides. A Disallow: / there blocks OAI-SearchBot. This is a common way a site drops out of ChatGPT search without anyone deciding it should, often a staging rule that survived launch.
  2. If you add a group for OAI-SearchBot, it stops reading the * group. Any rules you rely on in the catch-all group, such as keeping crawlers out of /admin/ or /checkout/, no longer apply to OAI-SearchBot unless you repeat them in its own group.

What should my robots.txt say?

Open your current file first and note what it already does. The examples below are starting points, not a file to paste over yours. Each one repeats an illustrative private path (/admin/) to show that rules from the catch-all group must be copied into named groups if you still want them.

Allow ChatGPT search and allow training:

User-agent: OAI-SearchBot
Disallow: /admin/

User-agent: GPTBot
Disallow: /admin/

User-agent: *
Disallow: /admin/

A named group with no blocking rule other than /admin/ allows everything else. You could also leave both crawlers to the * group, as long as that group does not block them.

Allow ChatGPT search, but not training:

User-agent: OAI-SearchBot
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Stay out of ChatGPT search answers:

User-agent: OAI-SearchBot
Disallow: /

OpenAI notes that if you allow both crawlers, it “may use the results from just one crawl for both use cases”. That does not change what each setting means.

OpenAI says “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” Wait a day before you test the effect.

For Anthropic, Perplexity and Google’s crawlers, see our AI bot permissions in robots.txt guide.

How do I check whether OpenAI can actually reach my site?

Work through these in order. Each step answers a different question, and a pass at one step does not prove the next.

1. Read the robots.txt file

Open yoursite.com/robots.txt. Find the group that applies to OAI-SearchBot: its own group if one names it, otherwise the User-agent: * group. Check that group for Disallow: / or a rule covering the pages you care about. Do the same for GPTBot and check the result matches what you intended.

2. Check the live response

curl -sI -A "OAI-SearchBot" https://yoursite.com/

A 200 means a request carrying that user agent string, from your machine, got through. A 403, a 429 or a challenge page means something in front of the site turned it away.

This is not proof that OpenAI’s crawler gets the same answer. Anyone can send that user agent string, and many firewalls judge the IP address as well.

3. Check the firewall or CDN

Look for bot-protection, “block AI bots” or rate-limit rules at your host or CDN. Several CDNs offer one-click AI crawler blocking that sits outside robots.txt entirely, so the file can say allow while the network says no.

4. Look for the real crawler in your logs

OpenAI publishes the IP addresses each crawler uses, for example https://openai.com/searchbot.json for OAI-SearchBot. Search your server or CDN logs for requests whose user agent contains OAI-SearchBot and check that they come from those ranges and received a 200. A request with the right name from the wrong address is not OpenAI.

5. Check search eligibility, then actual answers

Steps 1 to 4 tell you whether OAI-SearchBot can reach your pages. They do not tell you whether ChatGPT uses them. To see that, ask ChatGPT the questions your customers ask, with search on, and note whether you are named or linked. Our guide on how to measure your AI visibility without paying for a tool sets out a simple log for doing this properly.

Can SearchScore check this for me?

The free ChatGPT visibility checker reads your robots.txt and reports access for GPTBot, OAI-SearchBot and ChatGPT-User, alongside other signals that affect whether ChatGPT can find you. The free SearchScore audit covers the other AI crawlers too, with a Fix-List of what to change. Both read your robots.txt and pages. Neither can see your firewall’s decisions about OpenAI’s real IP addresses, so step 4 above is still worth doing.

Where this fits

Crawler access is the first gate. Once OAI-SearchBot can reach your pages, the questions become whether those pages answer what buyers ask and whether ChatGPT treats them as a source. Our guide on how ChatGPT cites websites covers that next step. If pages have dropped out of search more generally, these four technical checks are the place to start.

Sources

  1. OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots (checked 12 November 2026)
  2. IETF, RFC 9309 Robots Exclusion Protocol, section 2.2.1: https://www.rfc-editor.org/rfc/rfc9309.html (checked 12 November 2026)
  3. Google Search Central, How Google interprets the robots.txt specification: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt (checked 12 November 2026)
  4. SearchScore, AI Crawler Policy Study 2026 (measured 18 August 2026): https://searchscore.io/research/ai-crawler-policy-2026/
  5. SearchScore, AI crawler policy re-measure (measured 20 October 2026): https://searchscore.io/research/ai-crawler-policy-2026-10/

Platform facts on this page were last checked against the sources above on 12 November 2026. OpenAI updates its crawler page from time to time; re-check it before relying on the user agent details.

Frequently asked questions

Is GPTBot the crawler ChatGPT search uses?

No. OpenAI documents GPTBot as the crawler that collects content which may be used to train its generative AI foundation models. The crawler OpenAI says is used to surface websites in ChatGPT's search features is OAI-SearchBot.

If I block GPTBot, will I disappear from ChatGPT search?

Not because of that rule alone. OpenAI says each crawler setting is independent, so you can disallow GPTBot and still allow OAI-SearchBot. Check the rest of the file too: if OAI-SearchBot has no group of its own, a catch-all User-agent: * group decides what it may crawl.

Does robots.txt control ChatGPT-User?

Not reliably. OpenAI says ChatGPT-User visits a page when a user's request in ChatGPT or a custom GPT needs it, that robots.txt rules may not apply because the action is user-initiated, and that ChatGPT-User is not used to decide whether content may appear in search. OpenAI points site owners to OAI-SearchBot for search opt-outs.

How long does a robots.txt change take to affect ChatGPT search?

OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. That is the time for the new setting to take effect, not a promise that your pages will then be shown for any particular question.

Does a 200 response to curl with the OAI-SearchBot user agent prove OpenAI can crawl my site?

No. It shows that a request with that user agent string, from your own IP address, got through. Firewalls and CDNs can treat real crawler traffic differently, often by IP address. Check your server or CDN logs for requests from the IP ranges OpenAI publishes.

Part of Pillar Article - see all guides in this series →