How to check if AI crawlers can access your website
To check if AI crawlers can access your website, you need two checks, not one: what your robots.txt file says, and what your server actually returns when an AI crawler connects. In our scan of 102 business sites in August 2026, 13 of them (12.7%) turned an AI crawler away with an HTTP 403 or 429 response even though their robots.txt said the crawler was welcome. The robots.txt said yes; the CDN said no. AuditLamp scan corpus, 102 sites, August 2026. Edge-block study: auditlamp.com/learn/notes/one-in-eight-scanned-sites-blocks-ai-at-the-edge.
Why does your robots.txt not tell the full story?
robots.txt states your policy. It does not measure your actual response. Your robots.txt can say "AI crawlers are welcome" and your CDN can simultaneously return a 403 to every request that identifies as an AI bot. The CDN acts before your page is served, and it does not read the robots file first. It checks the requesting user-agent against its own rules.
This split happens most often with Cloudflare's bot management settings. Enabling the "AI scrapers and crawlers" block or the "bot fight mode" toggle in the Cloudflare dashboard blocks AI crawlers at the network layer. Your robots.txt stays permissive; your site stops being crawled. Standard SEO audits and robots.txt checkers will not catch this -- they read the file, not the server response.
Which AI crawlers should you check for?
For AI search visibility -- appearing in ChatGPT, Perplexity, and Claude answers -- the crawlers that matter are the retrieval bots: OAI-SearchBot (OpenAI web search), Claude-User (Anthropic browsing), and Perplexity-User. These are the crawlers that fetch live pages to build answers. Blocking them removes your site from AI search results.
Training bots (GPTBot, ClaudeBot, Google-Extended) are a separate decision. Blocking them keeps your content out of future model training. Blocking GPTBot does not block OAI-SearchBot; they are different user-agents with different purposes and different robots.txt rules.
How do you run the check yourself?
Three methods, in order of reliability:
- AuditLamp scan: run a live AI crawler access check for your URL. We send HTTP requests that identify as OAI-SearchBot, Claude-User, and Perplexity-User and report the exact response code each one receives from your server and CDN. This catches the edge-block pattern that robots.txt checks miss.
- Manual robots.txt check: Open yourdomain.com/robots.txt in a browser. Search for the user-agent strings of the crawlers you care about (GPTBot, OAI-SearchBot, ClaudeBot). A Disallow: / rule blocks them; a missing entry or Allow: / permits them. This only tells you your stated policy, not your actual access.
- CDN audit: Log in to your CDN or hosting control panel. Look for security or bot settings labelled "AI bots," "known bots," "bot fight mode," or "managed challenge." If any of these are enabled for AI crawlers specifically, they may be overriding your robots.txt welcome.
What do you do if AI crawlers are blocked?
If the block is in robots.txt: remove the Disallow directive for the crawlers you want to allow and republish. Changes take effect on the next crawl. For how to fix a robots.txt that blocks AI crawlers, including the exact directive syntax for each major crawler, see our fix guide.
If the block is at the CDN: go to your CDN's bot management settings and look for rules that target AI crawlers. In Cloudflare, the relevant settings are under Security > Bots. In other providers, look for "bot mitigation," "good bot allowlisting," or "known bots." Allow the specific user-agents you want to permit; do not blanket-disable bot protection entirely.
For a full technical picture including robots.txt, CDN response, and JavaScript rendering access, run the full technical SEO and robots.txt audit.