Google says that Cloudflare robots directive does nothing. What should you check?
Trade press covered Google saying a Cloudflare "content signals" robots.txt directive has no effect on Google. That is a secondary citation. Useful. Incomplete.
What owners need is not another recap of the news. They need a free way to see what still fails on their site.
What we measure (ours)
On the free AuditLamp Visibility Scan we probe named AI crawlers and report access problems in plain language. The focused tool is auditlamp.com. The full site scan (up to 150 pages, 164 graded checks) includes those access checks with the rest of the build.
We also studied crawler blocking patterns and published auditlamp.com.
In a 60-site small-business corpus from our production data (analysis dated 2026-07-15), robots-level search-AI disallow of the kind people imagine was 0 of 60 classifiable domains, while about 18.3% still rejected AI crawler user-agents at the edge. Full method notes live with the corpus write-up for slate post 17. Edge defaults matter even when robots look friendly.
What to do if you use Cloudflare (or any CDN)
- Treat AI bot toggles and bot fight modes as product decisions, not invisible defaults.
- Run the free scan on the live hostname customers use (www vs apex, the real CDN front).
- Read AI crawler rows first. Fix access before you rewrite content for "GEO."
- Re-scan after you change rules. Keep the report if you work with an agency.
What this page is not
- Not legal advice about training opt-outs.
- Not a promise that allowing a bot makes ChatGPT cite you.
- Not a substitute for reading Google's own documentation on crawlers and robots. When Google comments on a vendor directive, read the primary source. We cite news coverage as secondary context only.
Why we wrote it
Radar keeps surfacing the Cloudflare directive story. Our corpus and product already answer the next question: what is true on my domain right now? That is the gap.
Run the free scan: auditlamp.com
Related: auditlamp.com · auditlamp.com · auditlamp.com