What percentage of websites block AI crawlers? (2026 data)
77.9% of the top 1,000 most-visited websites have no AI crawler block at all. AuditLamp measured 653 reachable sites from the Tranco top-1,000 list (crawled 2026-07-05); 22.1% block at least one AI crawler via robots.txt. Training bots are blocked far more than retrieval crawlers: GPTBot (OpenAI training) was blocked by 16.7% of sites, while OAI-SearchBot (the crawler that actually builds ChatGPT answers) was blocked by only 9.2%.
Which AI crawlers do sites block most?
Across the 653 measured sites, block rates by crawler were: CCBot 18.7%, Bytespider 17.6%, GPTBot 16.7%, Google-Extended 15.0%, PerplexityBot 14.1%, Claude-User 9.7%, OAI-SearchBot 9.2%, Bingbot 2.3%, Googlebot 2.0%. The pattern is consistent: training bots (which scrape content to build AI models) are blocked about twice as often as retrieval bots (which fetch pages to cite them in live answers).
Does blocking a training bot also block AI citations?
No. Training bots and retrieval bots are separate crawlers with separate user-agent strings and separate robots.txt rules. Blocking GPTBot in robots.txt tells OpenAI not to use your content to train its models. It has no effect on OAI-SearchBot, which is the crawler OpenAI uses when a user asks ChatGPT a question and it needs to fetch live pages to build an answer. A site can block GPTBot (training) and remain fully citable by OAI-SearchBot (retrieval). They are independent decisions.
What about CDN-level blocking that robots.txt does not show?
robots.txt reports stated policy, not measured access. In a separate AuditLamp corpus of 60 small-business sites (measured 2026-07-15), 18.3% had CDN or firewall rules that sent AI crawler user-agents a 404 or challenge response on pages that loaded normally in a browser. These sites have compliant robots.txt files but the CDN blocks the bot before the server is reached. Standard 404 monitoring does not catch these because the monitoring tools watch Googlebot, not AI crawlers specifically.
How to check if your site blocks AI crawlers
Run AuditLamp on your domain. We probe each major retrieval bot user-agent (OAI-SearchBot, PerplexityBot, Claude-User) and show the actual HTTP response each one receives from your server and CDN, separate from what robots.txt states. The check is free.
Methodology
AuditLamp fetched the Tranco top-1,000 most-visited global domains on 2026-07-05. Of 1,000 domains attempted, 653 were reachable (returned a 2xx or 3xx that resolved). Each site's robots.txt was parsed for explicit Allow/Disallow directives against each crawler user-agent. Block rates are the share of the 653 reachable sites with at least one Disallow rule matching that user-agent. The small-business corpus (60 sites) was a separate convenience sample measured 2026-07-15 and is not representative of the top-1,000.
Full methodology and the per-bot breakdown chart: AuditLamp AI-crawler blocking study.