Your robots say yes. Your CDN may still block AI.
We scanned 60 real small-business sites in our corpus. Not one of the classifiable domains blocked ChatGPT-class search in robots.txt by disallowing the search AI bots we could score that way. About 1 in 5 (18.3%) still turned AI crawler user-agents away at the edge. For 13.3%, that edge behavior showed up with no clear opt-out signal in a stated policy.
If you only checked robots.txt, you would think AI was welcome. The machines never got in.
How we know
This is not a vibes post. On 2026-07-15 we pulled scan rows from our own production data (D1) for a 60-site small-business corpus and classified robots and edge behavior for AI crawlers. Full method notes live in our internal corpus write-up for slate post 17. We do not claim a robots-level "X% invisible to ChatGPT by accident" story: robots-level search-bot disallow in that corpus was 0 of 60. The real risk we measured is different.
We also do not probe every possible AI search user-agent at the network layer in every product build. Where we cannot measure a specific bot, we say so. The free AuditLamp scan does test major AI crawlers by name on your site (GPTBot and related probes in the product matrix). Run it if you want the check on your domain, not a blog average.
What this means if you run a real business site
- Robots can look friendly while the CDN is rude. A default firewall or "AI bot" toggle can reject crawlers your robots file never mentioned.
- Small sites drift into the block. Agencies and hosts ship templates. You inherit the block without a decision.
- "Invisible to ChatGPT" is not always a content problem. Sometimes it is access. Fix access before you rewrite the blog.
What to do this afternoon
- Open your host or CDN security settings. Look for AI crawler, bot fight, or managed challenge rules that apply to non-browser user-agents.
- Check robots.txt for Disallow rules on GPTBot and other AI crawlers. Our corpus rarely saw intentional robots blocks on small sites. Still verify yours.
- Run a free Visibility Scan at auditlamp.com. The AI crawler access checks are part of the same engine we use on our studies. Worst problems first, plain language.
- If you use Cloudflare or similar, treat AI toggles as a product decision, not a default you ignore.
What we are not saying
- We are not saying every site must allow every bot. You may have a reason to block. Say it on purpose.
- We are not selling a magic sentence that makes ChatGPT cite you.
- We are not reporting third-party "most blocked bot" leaderboards as if they were our sample. When we cite outside work, we label it.
Method, short
N = 60 small-business domains with classifiable scan rows in our production corpus as of the 2026-07-15 analysis. Robots-level search-class AI disallow among classifiable rows: 0. Edge-level AI UA rejection: 18.3%. Edge rejection without a clear stated opt-out signal: 13.3%. Details and caveats: internal corpus study notes for post 17; product probes document which bot UAs we send.
Next
If your scan shows an AI crawler fail, fix the access path, re-scan, then worry about answer blocks and schema. Access first.
Free whole-site Visibility Scan (up to 150 pages): auditlamp.com
Related: auditlamp.com ยท auditlamp.com