The setting that blocks AI training bots can now catch Googlebot in the same net
We wrote earlier this week about GPTBot becoming the most blocked crawler on the web, and how most of that blocking happens by accident, through defaults nobody chose on purpose. Here is the sharper edge of the same story: the accident can now include Google itself.
Search Engine Journal reports that Cloudflare now lets all sites manage AI crawlers under three classifications, Search, Agent, and Training, with new defaults arriving September 15 that can end up blocking Googlebot for sites that block training (searchenginejournal.com). The mechanics matter less than the outcome: a switch that reads as "protect my content from AI training" can, in the wrong configuration, interfere with the crawler that puts you in ordinary Google Search results.
The deadline is real. As Search Engine Roundtable reported, Cloudflare set September 15, 2026 for new content-signal defaults, and Cloudflare is used by approximately 21.3% of all websites as of January 2026 (seroundtable.com). If your site runs through Cloudflare, and roughly one site in five does, this is a calendar item, not a think piece.
Why this failure mode is so quiet
Nothing visibly breaks. Your site loads fine for you and every customer who visits. The only party turned away at the door is a crawler you never see. The first human-visible symptom is a slow bleed in search traffic weeks later, at which point the cause is buried under every other explanation: seasonality, competitors, an algorithm update, the weather.
This is why we keep hammering one principle: blocking decisions should be decisions. Search Engine Journal's own advice column this month framed the real question well, asking whether you should block AI crawlers or measure their value first, noting that AI bots do consume server resources and scrape content, but blocking them risks invisibility in LLM answers (searchenginejournal.com). There are legitimate reasons to block. There are no legitimate reasons to block by accident.
The fifteen-minute audit
If you use Cloudflare: log in, find the AI crawler and content-signal settings, and read what is actually enabled. Note what you decided and the date. Before September 15, check once more, because the defaults are changing underneath you, per the reporting above.
If you do not know what your site is telling crawlers: that is precisely the visibility gap AuditLamp exists to close. The free scan reads your site the way Google and the AI answer engines actually read it, including whether the crawlers that matter can get in at all, and tells you in plain language what they see today. No email required to get your diagnosis.
Sources: Search Engine Journal, "Cloudflare's AI Crawler Rules Can Block Googlebot," July 2, 2026 (searchenginejournal.com). Search Engine Roundtable, "Google: Cloudflare Content Signals Robots.txt Directive Has No Effects," July 6, 2026 (seroundtable.com). Search Engine Journal, Ask An SEO on blocking vs. measuring AI crawlers, July 2, 2026 (searchenginejournal.com).