Who blocks the AI crawlers?

We fetched the robots.txt of the 1,000 most-visited websites and read their rules for every major AI and classic crawler. The finding that matters: the web has stopped treating "AI bots" as one thing. Sites now block a company's training crawler while waving through its search crawler. OpenAI's GPTBot is blocked nearly twice as often as its OAI-SearchBot. The policy is becoming: cite me, don't train me.

The training / retrieval split

Take each AI company's two bots side by side. The training crawler (builds the model's corpus) is refused far more often than the retrieval crawler (fetches your page live to answer a user and, often, link them to you). That gap is a deliberate editorial choice, and it is now written in machine-readable robots.txt.

Every crawler, ranked by how often it is blocked

Measured against the 653 sites that answered a plain request on their bare domain. Red bars are AI training crawlers, grey are AI retrieval/search crawlers, black is classic web search.

Common Crawl takes the most fire

The single most-blocked crawler is not any AI company's — it is CCBot, the open Common Crawl dataset, at 18.7%. Blocking one dataset is the highest-leverage move a publisher can make, because Common Crawl feeds many downstream models at once. Meanwhile classic search is sacred: only 2.0% turn away Googlebot. A decade of "don't touch robots.txt or you vanish from Google" still governs behavior. And the loud AI-blocking movement is still a minority: 77.9% of these sites block no AI crawler at all.

Does anyone publish llms.txt?

11.9% of the reachable top-1,000 serve an llms.txt file — Cloudflare, GitHub, Azure, WordPress, Adobe, Samsung, Dropbox, PayPal, Shopify among them. Read that number carefully: this is the tech-forward head of the web, and adoption in the long tail is near zero. llms.txt is not a Google requirement and no engine documents it as a ranking input. We report its adoption as a curiosity, not a to-do.

Get the data

The full per-domain results — every one of the reachable sites, every bot, allow or block — are in one CSV. Dated, methodology below, free to use with attribution.

Method

• Sample: the Tranco top 1,000 (list JZ2VY, dated 2026-07-01), a research-grade, citable ranking that averages five providers over 30 days.
• Requests: exactly two per domain — /robots.txt and /llms.txt. No page crawling, no recursion. An honest, self-identifying user-agent (AuditLampResearch/1.0) with a link back to this page.
• Reachable: 653 of 1,000 returned a response to a plain HTTPS request on the bare domain. The rest are CDN / API / infrastructure hostnames or refused the connection, and are excluded rather than counted as failures.
• "Blocked": the site's robots.txt disallows the homepage / for that user-agent token, evaluated by longest-match RFC 9309 — the same evaluator the AuditLamp engine uses on live audits.
• Limits: robots.txt is a request, not a fence — this is stated policy, not measured compliance. The sample is the head of the web, not the whole web. And it is a single dated snapshot; the value is in re-running it and watching the trend.