Note

We scanned 60 small-business sites. Almost none had decided what AI can see.

The short answer: we pulled the most recent audit of every real business site our engine has scanned, 60 domains, and checked who actually controls what AI crawlers can read. Not one blocks an AI search crawler in robots.txt, and 95% have no AI crawler policy at all. But roughly 1 in 5 of those same sites turned away a documented AI crawler at the network edge, a firewall setting most owners never chose. The big end of the web is deciding its AI visibility on purpose. Small businesses are letting a CDN default decide for them.

What we measured

Our audit engine keeps the full result of every scan it runs. On July 15, 2026 we took the latest scan of each distinct business domain in that corpus, excluded our own sites, test domains, and verification scans, and kept the 60 that carry the modern per-bot checks. For each site we looked at two layers:

• The policy layer: what the site's robots.txt says to each documented AI crawler, split the way the vendors document them: search-class crawlers that decide whether you appear in AI answers, and training-class crawlers that only read for model training.
• The door layer: what actually happens when a documented AI crawler user-agent requests the page, compared with a normal browser making the same request.

The numbers

• 0 of 60 block a search-class AI crawler in robots.txt. Zero. Nobody wrote a rule that makes them invisible to AI search on purpose.
• 3 of 60 (5%) opt out of training crawlers only, and all three carry the identical Cloudflare-managed six-bot block. That is a deliberate, documented choice, and it does not touch AI search visibility.
• 57 of 60 (95%) have no AI crawler rules at all. Open by default, not by decision.
• 11 of 60 (18%) refused a documented AI-bot user-agent at the firewall with a 403 or 429 while a normal browser sailed through on the same page.
• 8 of 60 (13%) did that with no stated opt-out anywhere on the site. We cannot see intent from the outside, so we will not call it an accident. We will say what the data says: blocked, with no sign anyone chose it.
• 7 of 60 (12%) turned away a search-class crawler that can cite them in answers people actually read.

Honest caveats, up front: our probes come from a datacenter IP, so an edge refusal proves an active bot-management rule exists, not that the verified vendor bot always fails. Sixty sites is a small corpus, self-selected toward owners who cared enough to run a scan, and skewed to US service businesses. Treat the edge numbers as an upper bound on the block and a lower bound on the confusion.

The contrast that matters

We also run a standing study of the top 1,000 sites on the web. There the picture is reversed: 16.7% block GPTBot in robots.txt, 9.2% block OAI-SearchBot, and 5.8% block training crawlers while deliberately allowing retrieval. Those are written rules, reviewed by someone whose job is to decide.

That is the real story. It is not that small businesses are blocking ChatGPT by accident. It is that the businesses with the most to lose from a wrong default are the only ones not making a choice at all. A national publisher decided what AI can read. The plumber whose next customer asks an AI assistant for "a plumber near me who answers on weekends" has a setting somewhere in a CDN dashboard, chosen by nobody, deciding it instead.

What to do about it, this week

1. See your own doors. Run the free scan at auditlamp.com. It requests your pages the way the documented AI crawlers do and names which ones your site turns away, in plain language.
2. If a crawler is blocked and you never chose that, ask your host or CDN about their bot protection defaults. Products in the "bot fight" family ship with AI crawlers bundled into the block list.
3. Then decide on purpose. Blocking training crawlers is a legitimate choice some owners want. Blocking search-class crawlers makes you invisible in the places buyers now ask. Either is fine as a decision. Neither is fine as a default you have never seen.

We take our own medicine: today's fresh scan of auditlamp.com is report 514, scored 98/100 across 150 pages, public at auditlamp.com. The top-1000 numbers come from our live study, updated at auditlamp.com. The corpus method and queries are dated 2026-07-15 and reproducible.