One AI can cite your business while another is blindfolded. Here is the split.
The short answer: the big publishers now run two different doors for AI. They slam the door on the crawler that trains models for free, and hold the door open for the crawler that might send them a reader. When we checked the top 1,000 sites on 2026-07-05, 49 of them blocked OpenAI's training bot but allowed its search bot on purpose. Most small business owners have the reverse problem, by accident: they block the bot that could recommend them and never know it.
There is no single "AI crawler"
When people say "should I block AI," they picture one switch. There isn't one. Every AI company runs at least two separate crawlers with different names, and they do different jobs:
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Bytespider) collect text to build and improve models. Blocking them keeps your words out of the training set. It does not, by itself, keep you out of the answers.
- Retrieval crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) fetch pages at answer time so the assistant can cite a live source. If you block these, you cannot be cited, full stop.
These are separate lines in your robots.txt. You can allow one and block the other. The biggest sites on the web already do.
The data: publishers are drawing the line on purpose
From our own crawl of the top 1,000 (2026-07-05, against 653 reachable sites):
- OpenAI's training bot (GPTBot) is blocked by 16.7%. Its search bot (OAI-SearchBot) is blocked by 9.2%. Nearly twice the block rate for the one that trains.
- Anthropic shows the same gap: ClaudeBot 16.1%, Claude-User 10.3%.
- 49 sites block GPTBot and allow OAI-SearchBot. Named: LinkedIn, Yahoo, Medium, Forbes, eBay.
- 38 sites (5.8%) block at least one training crawler while allowing every retrieval crawler we tested. A deliberate "cite me, do not train me" stance.
These publishers have lawyers and licensing deals. The stance makes sense for them: do not give the model away for free, but stay quotable so you still get the click.
The small-business version of this is usually an accident
Here is the trap. A local business almost never has a "cite me, do not train me" policy. What it has is a robots.txt it never opened, often a template from its hosting setup or an old SEO plugin, that blocks a bot nobody remembers deciding to block. Sometimes that is a retrieval bot. Which means an AI assistant that wanted to recommend the business could not read the page to do it.
You are not a publisher protecting a licensing deal. You want to be found. For you, the default should almost always be: let the retrieval crawlers in.
Check yours in 60 seconds
- Open `yourdomain.com/robots.txt` in a browser.
- Look for any `Disallow: /` that sits under `User-agent: OAI-SearchBot`, `User-agent: ChatGPT-User`, `User-agent: Claude-User`, or `User-agent: PerplexityBot`.
- If you find one, that is an AI assistant you have locked out of citing you.
If reading robots.txt syntax is not your idea of a good afternoon, that is fair. It is finicky and the longest-match rules are easy to get wrong.
An honest caveat
robots.txt is a request, not a wall. It states your policy. It does not physically stop a crawler, and a few have been publicly accused of ignoring it. Allowing the retrieval bots does not guarantee a citation either. It removes the most common self-inflicted reason you would never get one. We sell measurement, not magic. There is no single line you paste to make AI love you.
Let the scan read your doors for you
Paste your URL into the AuditLamp scan. It reads your robots.txt with the same RFC 9309 evaluator we used for the study and tells you, in plain language, which AI crawlers can see you and which are shut out. The diagnosis is free and there is no email wall on it.