AI crawler optimization starts with one file. Google's documentation defines it: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." The same file controls the 15 AI bots we test, from GPTBot to PerplexityBot, and one careless wildcard can turn every AI answer engine away at your door. We check each bot by name and show you the exact rule responsible.
The AI answer engines send their own crawlers to decide what to read, index and cite. If your robots.txt is misconfigured, those crawlers are turned away before they can consider you as a source.
Free Visibility Scan: whole site, up to 150 pages, no email. Full Report is a one-time $10.
AI bots split into two types: retrieval crawlers that fetch your page when someone asks a live question (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User, DuckAssistBot), and training crawlers that build a model's base knowledge (GPTBot, ClaudeBot, Google-Extended, Amazonbot, Bytespider).
These two populations are independent controls in robots.txt. Blocking a training crawler does not remove you from an engine's answers. Blocking a retrieval crawler does: that bot can no longer fetch your page when a user asks something you should be answering.
Most sites that think they are "blocking AI" are only blocking training bots. A growing share, including many that never intended to disappear, have rules that block retrieval bots too. They are invisible to those engines for live queries, and they do not know it.
The full breakdown: fixing AI crawler access and GPTBot and robots.txt.
The free scan gives you the per-bot access matrix, your posture label, a scored verdict on retrieval-bot blocks with the exact robots.txt rule causing each problem. Need a corrected file right now? The free robots.txt generator and llms.txt generator build them. Monitor ($29 a month) rescans weekly, so a deployment that pushes a bad robots.txt rule gets caught. The umbrella view is the AI search optimization hub.
No. GPTBot is OpenAI's training crawler. OAI-SearchBot and ChatGPT-User are the retrieval crawlers that fetch your page when someone asks a live question in ChatGPT. These are separate robots.txt controls. Blocking GPTBot does not affect ChatGPT search results. Read more: GPTBot and robots.txt.
Not in traditional search. Blocking GPTBot, ClaudeBot or Google-Extended is a legitimate policy choice and does not affect Googlebot, Bingbot or retrieval-class AI bots. We say so plainly in the report; we do not pressure you to open training access.
No robots.txt means all crawlers are allowed by default. That is a valid posture. We report it accurately.
Yes. The report names the exact user-agent match and Disallow path that causes each block, so you know precisely what to change.
The robots.txt check applies site-wide. The JavaScript render check runs against both a desktop and mobile fetch of your homepage.
Free scan, no account. Every finding links to the vendor documentation that grounds it, and every verdict is derived from what we actually fetched.
15 bots by name · retrieval vs training separated · part of the 164-check audit