GPTBot isn't how ChatGPT finds you: where each AI engine's shortlist actually comes from

Ask ChatGPT which short-term insurer a South African family should use and a shortlist comes back: three or four names, stated as fact. If your brand is missing, the obvious first move is to open robots.txt and look for GPTBot. For this question, that's the wrong line to check.

This post asks where each of the four assistants gets the names on its shortlist, and which of your settings affect that. We answer only from what OpenAI, Anthropic, Google and Perplexity document on the record. We read every page on 23 September 2026. Crawler policies changed through 2025 and 2026, so treat this as a dated read.

A shortlist has two sources, and they answer to different inputs

An assistant can name a brand from what its model absorbed in training, or from pages it retrieves while answering. Memory reflects what the web said before a training cutoff. Retrieval reflects what a search layer can reach and rank today. Work that shifts one need not show up in the other.

Each engine decides for itself when to search. OpenAI's web search guide says "the model can choose to search the web or not based on the content of the input prompt." Google's grounding documentation says the model "analyzes the prompt and determines if a Google Search can improve the answer." Anthropic's web search tool documentation goes furthest. It lists questions about "specific organizations, people, or products that might have changed" among the triggers for a search, and says Claude answers directly when a request "draws on stable knowledge."

These pages describe the developer platforms, and the consumer apps may behave differently. Even so, whether a brand gets judged from memory or from a fresh fetch depends on the engine and on the question. That's why we treat this as four races, not one.

ChatGPT: OAI-SearchBot surfaces you, GPTBot trains the model

OpenAI's crawler documentation separates three agents. OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features," and sites that opt out of it are not shown in ChatGPT search answers. GPTBot is for training: "Disallowing GPTBot indicates a site's content should not be used in training." ChatGPT-User handles fetches that a person triggers in a conversation. Because a user starts those fetches, "robots.txt rules may not apply."

On the point that matters here, OpenAI is explicit: "Each setting is independent of the others." Its own example is a site that allows OAI-SearchBot and disallows GPTBot. Blocking GPTBot is a training decision. By OpenAI's description, it does not remove you from ChatGPT search. OpenAI adds that its systems can take about 24 hours to adjust after a robots.txt change.

Neither OpenAI page we read names the search index or partner behind ChatGPT search. We leave that question open rather than repeat second-hand accounts.

Claude: three bots on the record, no search index named

Anthropic's crawler help page, updated 7 April 2026, draws the same line. Blocking ClaudeBot "signals that the site's future materials should be excluded from our AI model training." Claude-SearchBot "navigates the web to improve search result quality for users," and blocking it "may reduce your site's visibility and accuracy in user search results." Claude-User may visit sites when people ask Claude questions, and blocking it "may reduce your site's visibility for user-directed web search." The page says all three honour robots.txt.

Neither that page nor the web search tool documentation names the provider or index behind Claude's search results. Anthropic hasn't documented it on the record.

Google's AI features run on the Search index, and Gemini has its own token

Of the four, Google is the plainest. Its AI features page, last updated 10 December 2025, says that to appear as a supporting link in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet." It adds: "There are no additional technical requirements." The controls are the usual Search ones: nosnippet, data-nosnippet, max-snippet and noindex.

Elizabeth Reid, Google's VP and Head of Search, described the retrieval step at the AI Mode launch in May 2025. AI Mode uses a query fan-out technique, "breaking down your question into subtopics and issuing a multitude of queries simultaneously on your behalf." The same post said AI Overviews was driving an "over 10% increase in usage of Google" for the types of queries that show them.

Gemini is where Google draws the line differently. Google's crawler documentation, updated 14 July 2026, says the Google-Extended token controls whether crawled content is used to train Gemini models and to ground answers in Gemini Apps, "providing content from the Google Search index to the model at prompt time", and that it "does not impact a site's inclusion in Google Search." So the Search index feeds AI Overviews and AI Mode, while a single robots.txt token covers both training and grounding in the Gemini app. On ChatGPT, those jobs belong to separate bots.

Perplexity: the search bot is not a training bot

Perplexity's bot documentation describes PerplexityBot as "designed to surface and link websites in search results on Perplexity," and says it isn't used for AI model training. Perplexity-User supports user actions and "generally ignores robots.txt rules," because a person started the fetch. Perplexity says robots.txt changes can take up to 24 hours to register. The page doesn't say whether Perplexity uses any index other than its own crawl.

Most AI crawling is for training, which is why GPTBot gets the attention

Cloudflare's analysis of traffic on its network, published 29 August 2025, found that over the 12 months to July 2025, 80% of AI crawling was for training, 18% was for search and 2% was for user actions. It also counted pages crawled per referral in July 2025: 38,065.7 for Anthropic and about 1,091 for OpenAI.

On Cloudflare's numbers, most of the AI crawling a site sees is for training, and very little crawling of any kind comes back as visits. The search-facing agents make up a smaller share of the traffic, but they are the ones tied to appearing in a live answer.

What a crawler verdict can and cannot tell you

The SA AI Visibility Index reads answers "produced by an engine with live web search enabled, the same surface a real user sees, not the model's training memory," according to its methodology. For the crawler read, the Index fetches each tracked site as a normal browser and as GPTBot, ClaudeBot and PerplexityBot. It reports a 403 as a hard block only when a second fetch reproduces it, and scopes the block to the bots it affects. It checks Google-Extended in robots.txt only, because Google-Extended is a permission token, not a live crawler.

By their operators' own descriptions, two of those three agents are training crawlers. A GPTBot verdict tells you whether your server lets GPTBot in. It tells you nothing about OAI-SearchBot, which you need to check separately.

The Index's methodology sets the limit itself: "The crawler verdict is presented beside share-of-mention, never as its explanation. Absence from AI answers is multi-causal; the Index claims a cause only where it is directly measured." Crawler access is a precondition you can test. It does not explain why an engine named you or left you out.

What to actually do about it

• Read robots.txt by token, not by company. OAI-SearchBot, Claude-SearchBot and PerplexityBot are the search-facing tokens. GPTBot and ClaudeBot are training tokens. Google-Extended covers Gemini training and grounding in the Gemini app.
• Make the training decision on its own. OpenAI documents GPTBot and OAI-SearchBot as independent settings, so blocking GPTBot tells you nothing about ChatGPT search.
• For AI Overviews and AI Mode, confirm that the pages you care about are indexed and eligible to show a snippet.
• Check what your server actually returns as well as what robots.txt says. After any change, wait about a day before you retest.
• Read each engine separately. If you check ChatGPT and stop there, you have measured a quarter of your visibility at most.

The Index's free crawler check fetches your homepage as AI crawlers and compares the response with what a normal browser gets.

We read every provider page cited here on 23 September 2026. These policies have changed before and will change again. This is a point-in-time read of what the providers say, not a guarantee of how any engine will answer next month.