Who blocks the AI crawlers?

We fetched the robots.txt of the 1,000 most-visited websites and read their rules for every major AI and classic crawler. The finding that matters: the web has stopped treating "AI bots" as one thing. Sites now block a company's training crawler while waving through its search crawler. OpenAI's GPTBot is blocked nearly twice as often as its OAI-SearchBot. The policy is becoming: cite me, don't train me.

Key findings from 653 sites in the Tranco top 1,000 (measured 2026-07-05): 18.7% block Common Crawl's CCBot — the most-blocked crawler of all. OpenAI's training bot GPTBot is blocked by 16.7% — nearly twice as often as its search bot OAI-SearchBot (9.2%). The same training-vs-retrieval split holds for Anthropic: ClaudeBot (training) is blocked by 16.1% vs. Claude-User (retrieval) at 10.3%. PerplexityBot is blocked by 14.1%. Classic search crawlers are nearly untouched — Bingbot 2.3%, Googlebot 2.0%. The majority — 77.9% of these sites — block no AI crawler at all.

Cite this data: AuditLamp team. Who blocks the AI crawlers? robots.txt policy across the Tranco top 1,000. Measured 2026-07-05, published 2026-07-10. auditlamp.com/study/ai-crawler-blocking. Free to use with attribution.

The training / retrieval split

Take each AI company's two bots side by side. The training crawler (builds the model's corpus) is refused far more often than the retrieval crawler (fetches your page live to answer a user and, often, link them to you). That gap is a deliberate editorial choice, and it is now written in machine-readable robots.txt.

Every crawler, ranked by how often it is blocked

Measured against the 653 sites that answered a plain request on their bare domain. Red bars are AI training crawlers, grey are AI retrieval/search crawlers, black is classic web search.

Common Crawl takes the most fire

The single most-blocked crawler is not any AI company's. It is CCBot, the open Common Crawl dataset, at 18.7%. Blocking one dataset is the highest-leverage move a publisher can make, because Common Crawl feeds many downstream models at once. Meanwhile classic search is sacred: only 2.0% turn away Googlebot. A decade of "don't touch robots.txt or you vanish from Google" still governs behavior. And the loud AI-blocking movement is still a minority: 77.9% of these sites block no AI crawler at all.

Does anyone publish llms.txt?

11.9% of the reachable top-1,000 serve an llms.txt file: Cloudflare, GitHub, Azure, WordPress, Adobe, Samsung, Dropbox, PayPal, Shopify among them. Read that number carefully: this is the tech-forward head of the web, and adoption in the long tail is near zero. llms.txt is not a Google requirement and no engine documents it as a ranking input. We report its adoption as a curiosity, not a to-do.

Get the data

The full per-domain results are in one CSV: every reachable site, every bot, allow or block. Dated, methodology below, free to use with attribution.

Cite this study

AuditLamp team. Who blocks the AI crawlers? robots.txt policy across the Tranco top 1,000. Measured 2026-07-05, published 2026-07-10. auditlamp.com/study/ai-crawler-blocking. Free to use with attribution.

Method

• Sample: the Tranco top 1,000 (list JZ2VY, dated 2026-07-01), a research-grade, citable ranking that averages five providers over 30 days.
• Requests: exactly two per domain: /robots.txt and /llms.txt. No page crawling, no recursion. An honest, self-identifying user-agent (AuditLampResearch/1.0) with a link back to this page.
• Reachable: 653 of 1,000 returned a response to a plain HTTPS request on the bare domain. The rest are CDN / API / infrastructure hostnames or refused the connection, and are excluded rather than counted as failures.
• "Blocked": the site's robots.txt disallows the homepage / for that user-agent token, evaluated by longest-match RFC 9309, the same evaluator the AuditLamp engine uses on live audits.
• Limits: robots.txt is a request, not a fence: this is stated policy, not measured compliance. The sample is the head of the web, not the whole web. And it is a single dated snapshot; the value is in re-running it and watching the trend.

AI crawler access: what these numbers mean for your site

AI crawler access is whether the bots that fetch sources for ChatGPT, Perplexity, Gemini, and Copilot can reach your pages at all. Access is the gate: a page that blocks PerplexityBot cannot be cited by Perplexity regardless of its content quality. These rates show the distribution of access choices across the most-visited sites. If you block GPTBot but allow OAI-SearchBot, you are making the same "cite me, don't train me" policy choice 49 sites in this sample already made. Check your own site's AI crawler access with the free crawler access tool.