Note

Who actually blocks the AI crawlers? We checked the 1,000 biggest sites on the web.

The short answer: the most-blocked crawler on the web's front page is not OpenAI, Anthropic, or Google. It is Common Crawl, blocked by 18.7% of the biggest sites. Google's own crawler is blocked by 2.0%. We ran the check ourselves on 2026-07-05 across the 1,000 most-visited domains, and the full table is below. You can steal any number in this post. We would rather be cited than clicked.

What we did, stated plainly

We took the Tranco top 1,000 (a public, manipulation-resistant ranking of the most-visited sites, list JZ2VY dated 2026-07-01) and fetched two files from each: `/robots.txt` and `/llms.txt`. We evaluated every AI crawler's access with our own engine's RFC 9309 rules, the same evaluator that runs inside a normal AuditLamp scan.

653 of the 1,000 answered a plain request. The other 347 are CDN and API hostnames (`gstatic.com`, `akamai.net`, `googleapis.com`) that do not serve a real homepage. We excluded them rather than count them as failures. Every rate below is against those 653 reachable sites, which is the conservative denominator. Against only the 461 sites that publish a robots.txt, every block rate runs about 1.4 times higher.

The full block-rate table (our crawl, 2026-07-05)

| Crawler | What it feeds | Blocked | % of 653 | |---|---|---|---| | CCBot | Common Crawl (feeds many LLMs) | 122 | 18.7 | | Bytespider | ByteDance / TikTok (training) | 115 | 17.6 | | GPTBot | OpenAI (training) | 109 | 16.7 | | ClaudeBot | Anthropic (training) | 105 | 16.1 | | Meta-ExternalAgent | Meta AI (training) | 99 | 15.2 | | Google-Extended | Google Gemini (training) | 98 | 15.0 | | PerplexityBot | Perplexity (search index) | 92 | 14.1 | | Applebot-Extended | Apple Intelligence (training) | 81 | 12.4 | | Amazonbot | Amazon (Alexa / AI) | 78 | 11.9 | | Perplexity-User | Perplexity (user fetch) | 73 | 11.2 | | ChatGPT-User | ChatGPT (user fetch) | 71 | 10.9 | | Claude-User | Claude (user fetch) | 67 | 10.3 | | OAI-SearchBot | OpenAI (search index) | 60 | 9.2 | | Bingbot | Bing / Copilot | 15 | 2.3 | | Googlebot | Google Search | 13 | 2.0 |

Finding 1: the web now sorts AI bots by purpose

"AI crawler blocking" is not one decision anymore. The same company's bots get treated differently depending on what the bot is *for*. OpenAI's training crawler (GPTBot) is blocked almost twice as often as its search crawler (OAI-SearchBot): 16.7% versus 9.2%. Anthropic shows the same gap: ClaudeBot 16.1%, Claude-User 10.3%.

49 sites block GPTBot but explicitly allow OAI-SearchBot. Named ones: LinkedIn, Yahoo, Medium, Forbes, eBay. 38 sites (5.8%) block at least one AI *training* crawler while allowing *every* AI *retrieval* crawler we tested. That is a clean, machine-readable stance: cite me, do not train me. Publishers will refuse to be free training data but keep the door open to the bot that might send them a reader.

Finding 2: classic search is still sacred

Only 2.0% block Googlebot and 2.3% block Bingbot. A decade of "do not touch robots.txt or you will vanish from Google" still governs behavior. The AI wave has not changed it. Whatever a site does about AI, it protects its Google traffic first.

Finding 3: blocking Common Crawl is the most efficient move

CCBot is blocked more than any single AI company's bot. That is not a coincidence. Common Crawl is one open dataset that many downstream models are built from, so blocking it once is the cheapest way to keep your words out of many training sets at the same time. Publishers who understand the plumbing block the reservoir, not each faucet.

Finding 4: most sites still block nothing

77.9% (509 sites) block no AI crawler at all. The AI-blocking movement is loud, but even at the tech-forward head of the web it is still a minority. Most of the internet is wide open to AI, on purpose or by default.

The llms.txt reality check

11.9% of the reachable top 1,000 publish an `llms.txt` file. Adopters are exactly who you would guess: Cloudflare, GitHub, Azure, Shopify, WordPress, Adobe. Here is the honest part, because our whole product is honesty: `llms.txt` is not a Google requirement and no major AI provider has officially committed to reading it. Its adoption is a curiosity metric, not a compliance one. This 11.9% is the head of the web. In the long tail, adoption is near zero. We do not claim a whole-web figure from a top-1,000 sample.

What this means for your business

Two things an owner can act on today, both free:

1. Find out which crawler your site blocks by accident. Most small sites do not have a considered "cite me, do not train me" policy. They have a robots.txt they never wrote, often a hosting default, that quietly blocks the exact bot that could recommend them in an AI answer. That is the opposite of what you want.
2. Decide on purpose. If you want to be cited by AI assistants, the retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) must be allowed. Whether you allow the training bots is a values call, and a legitimate one. The point is to make it a choice, not an accident.

Honest limitations (they make this more credible, not less)

• The sample is the 1,000 most-visited sites. Tech-forward, not representative of the long tail. Every figure is "among the head of the web," never "the whole web."
• robots.txt is a request, not a fence. This measures stated policy, not compliance. Some crawlers have been publicly accused of ignoring it; we did not measure that here.
• "Blocked" means disallowed at the homepage `/` for that bot's token, by longest-match RFC 9309. A site could allow `/` and block a subfolder; we report the homepage decision.
• This is a 2026-07-05 snapshot. robots.txt changes daily. The value is in re-running it and showing the trend.

What is next

We built our own copy of the Common Crawl domain graph: 121 million domains, no third-party data API. The next edition of this study runs the same check across that full index instead of 1,000 sites, so we can report AI-blocking rates for the actual long tail and track the trend every quarter. When it is done, the numbers will be ours, dated, and free to cite.

Run it on your own site

You do not have to guess which crawlers your site allows. Paste your URL into the AuditLamp scan and it will tell you, in plain language, which AI crawlers can read you and which are locked out. The diagnosis is free and there is no email wall on it.