One in eight scanned sites blocks AI at the edge without knowing
We pulled the latest scan for each of the 105 business sites that have run our engine and compared two things: what the robots.txt says about AI crawlers, and what the site's server actually does when an AI crawler knocks. The two checks ran on 102 of the 105 sites. About one in eight (13 of 102, 12.7%) turned an AI bot away at the edge with an HTTP 403, even though their robots.txt said the bot was welcome. Only 2 sites blocked AI in robots.txt on purpose. The gap between those numbers is the story: most of this blocking is not a decision, it is a default nobody chose.
What is the difference between a robots block and an edge block?
Robots.txt is a text file at the top of your site that lists rules for crawlers. It is a request, and a well-behaved bot reads it before fetching anything. An edge block is different. It happens at the server or the CDN, before your page is ever served, when the firewall looks at who is asking and returns a 403 Forbidden. Your robots.txt can say "GPTBot, come in" while your CDN slams the door in the same bot's face. The bot never gets to read the welcome mat.
That is exactly what we found on all 13 sites. Every one of them had a robots.txt that allowed the AI crawlers. Every one of them then refused those crawlers at the edge.
How does this differ from the July crawler-blocking study?
In July we read the robots.txt of the 1,000 most-visited sites and published the results at auditlamp.com/study/ai-crawler-blocking. That study measured deliberate policy: 16.7% of reachable sites block OpenAI's GPTBot in robots.txt, 18.7% block Common Crawl, and only 2.0% block Googlebot. Those are choices, written in a file a person edited.
This is the other half of the picture, and it is quieter. Here the robots.txt says yes. The block lives one layer down, at the CDN, where an owner rarely looks. A July-style robots audit would score these 13 sites as fully open to AI. They are not. If you only check the file, you miss the block entirely.
| Layer | What it measures | Blocked in this corpus | |---|---|---| | robots.txt policy | The rule the owner wrote | 2 of 102 | | Edge reachability | What the server actually returns to an AI bot | 13 of 102 |
Why would a site block a bot it said it allowed?
Because the block ships in the box. Managed security products and CDNs increasingly include a bot-mitigation layer, and many bundle an "AI bots" or "known bots" toggle that is on by default or flipped on during setup. Cloudflare documents these controls in its bot management docs. When an agency or a host turns one on, the site inherits an AI block that no one wrote into robots.txt and no one revisited. Of the refusals we saw across these 13 sites, the large majority came back through Cloudflare, with a couple through nginx. The owner set the welcome mat by hand and never noticed the firewall contradicting it.
Does this really keep AI engines out?
We measure a user-agent block, not a network verdict, and we say so in every report. Our scanner probes from a datacenter IP, while real vendor bots crawl from published IP ranges (OpenAI lists GPTBot's ranges in its bot documentation). So a 403 here proves the site refuses a request that identifies itself as an AI bot. It does not prove the exact production crawler is stopped in every case. What it does prove is that the door is configured to reject that name, which is the opposite of what the robots.txt promised. When a site's stated policy and its actual behavior disagree, the safe reading is that the stricter one wins, and here the stricter one is the silent edge block.
What should an owner check this afternoon?
- Read your own two layers, not just one. Open robots.txt in a browser, then open your CDN or host security settings and look for anything labeled AI bots, bot fight, known bots, or managed challenge.
- If a rule targets non-browser user-agents, decide on purpose whether AI retrieval bots should be exempt. Blocking a training crawler while allowing a retrieval crawler is a legitimate choice. Blocking both by accident is not.
- Treat any AI toggle your host enabled as a decision to review, not a default to trust.
- Re-check after any host, theme, or security-plan change. This is where inherited blocks sneak in.
What we are not claiming
These 105 domains are self-selected. They are businesses and owners who chose to run our scan, skewed toward trades, local services, and small firms, so this is not a random sample of the web and we make no claim about a global rate. The check also ran on 102 of the 105 sites, not all of them, and the one-in-eight figure is within that measured group. We are not saying every site should allow every bot, and we are not selling a sentence that gets you cited. We are saying one specific, checkable thing: among ordinary business sites, a meaningful share tell AI "yes" in the file and "no" at the edge, and almost none of them know it.
The full list of what our engine tests is at auditlamp.com/what-we-check, and the AI crawler access check has its own tool page at auditlamp.com/tools/ai-crawler-access. How we grade every check, with the arithmetic shown, is at auditlamp.com/methodology.
See which layer your own site answers on
Your robots.txt and your CDN can disagree, and the disagreement is invisible until something reads both. Run a free Visibility Scan at auditlamp.com. It knocks on your site the way an AI engine does, reads your robots rules and your edge response together, and tells you plainly whether the two agree.