Note

What GPTBot actually fetches from your site

GPTBot is OpenAI's training crawler, and it is only one of four bots OpenAI documents. Its published job is to crawl "content that may be used in training our generative AI foundation models" (OpenAI crawler documentation). It fetches the public pages your robots.txt allows, and what it collects can end up inside future models, with no link back to you. The crawler that decides whether ChatGPT's search results show your site is a different bot entirely, and blocking one does not block the other.

Which OpenAI bot is which?

As of July 2026, OpenAI's crawler page lists four bots. Each has its own name for robots.txt, its own stated purpose, and its own published IP addresses.

• GPTBot crawls for model training. OpenAI states it "is used to crawl content that may be used in training our generative AI foundation models." It announces itself in your server logs with a user-agent string ending in "+openai.com", currently at version 1.4.
• OAI-SearchBot exists "to surface websites in search results in ChatGPT's search features." The documentation states it is not used to train models. This is the bot that matters if you want ChatGPT search to find and link your business.
• ChatGPT-User is not a bulk crawler. It fires for "certain user actions in ChatGPT and Custom GPTs": a real person asked about your page, and ChatGPT fetched it live to answer them.
• OAI-AdsBot checks pages submitted as ads in ChatGPT. OpenAI states its data "is not used to train generative AI foundation models."

Train or retrieve? Two different deals

Every OpenAI fetch is one of two transactions. A training fetch takes your words to help build the next model. If that ever pays you back, it pays diffusely, later, without credit. A retrieval fetch reads your page right now, to answer one specific person, and can put your name and your link in front of them. Those are different deals with different economics, and your robots.txt answers them separately, by bot name.

Most robots.txt files we read never made that decision. They either allow everything because nobody looked, or block everything because one plugin promised "AI protection." We wrote a plain-language guide with posture templates for small businesses, publishers, and SaaS docs at our AI bot policy page. The short version: block training, keep answers, or the reverse. Just decide on purpose.

Does blocking GPTBot hide you from ChatGPT?

No. The bots operate independently, and OpenAI's documentation is explicit that a site can disallow GPTBot for training while still allowing OAI-SearchBot for search. The damage we see runs the other way: blanket rules that block everything with AI in the name, often installed at the firewall in one click, cut off the retrieval bots along with the training one. That trades a content-policy stance for invisibility at the exact moment people ask ChatGPT who to hire or what to buy.

One honest caveat. Disallowing GPTBot governs OpenAI's crawling from that point forward. It does not recall anything already collected, and it says nothing to other companies' crawlers. Each of those needs its own line in the same file.

How do you check what GPTBot can fetch today?

1. Open yourdomain.com/robots.txt in a browser. It is a public file, and it is the same file every crawler reads.
2. Search it for GPTBot, OAI-SearchBot, and ChatGPT-User. A bot with no rule naming it, and no "User-agent: *" disallow covering it, is allowed by default.
3. Expect a delay after edits. OpenAI's documentation says it "can take ~24 hours from a site's robots.txt update for our systems to adjust."
4. Remember that robots.txt is a request, not a wall, and it is not the only gate. Your CDN or firewall can block a bot your robots.txt welcomes. We wrote up that failure mode in your robots say yes, the CDN may still block AI.
5. Distrust the label in your logs. Anyone can send GPTBot's user-agent string. Real GPTBot traffic comes from the IP ranges OpenAI publishes at openai.com/gptbot.json, one JSON file per bot.

What our scan reports

An AuditLamp scan reads your robots.txt the way the bots do and reports the posture per bot: who is allowed, who is blocked, and whether the block covers training bots, retrieval bots, or both. It also probes reachability from outside, because a permissive robots.txt over an edge block is a yes that behaves like a no. The checks are named in public: gate.ai_bot_access, gate.bot_reachability, and core.train_vs_retrieve, all listed on what we check.

What we will not tell you is that allowing OAI-SearchBot gets you cited. It makes you eligible, nothing more. No tool can promise placement in an AI answer, and the ones that do are selling the promise, not the plumbing.

Your site is telling OpenAI's four bots something right now, deliberately or by accident. To see the per-bot answer in plain language, worst problems first, run a free Visibility Scan at auditlamp.com.