AI crawler optimization starts with one file. Google's documentation defines it: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." The same file controls the 15 AI bots we test, from GPTBot to PerplexityBot, and one careless wildcard can turn every AI answer engine away at your door. We check each bot by name and show you the exact rule responsible.
Free category scores and selected findings. Full findings and export files: $10 once.
Up to 150 pages. No email required.
For the full AI search layer beyond crawler access, run the full AEO and AI crawler audit.
OAI-SearchBot
Claude-SearchBot
PerplexityBot
GPTBot
ClaudeBot
CCBot
The AI answer engines send their own crawlers to decide what to read, index and cite. If your robots.txt is misconfigured, those crawlers are turned away before they can consider you as a source.
AI bots split into two types: retrieval crawlers that fetch your page when someone asks a live question (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User, DuckAssistBot), and training crawlers that build a model's base knowledge (GPTBot, ClaudeBot, Google-Extended, Amazonbot, Bytespider).
These two populations are independent controls in robots.txt. Blocking a training crawler does not remove you from an engine's answers. Blocking a retrieval crawler does: that bot can no longer fetch your page when a user asks something you should be answering.
Training and search rules need separate review. A site can block training crawlers while allowing search crawlers, or unintentionally restrict the search crawlers it wanted to reach. Those rules can restrict crawler access. They do not, by themselves, establish whether a business appears in a live answer.
The full breakdown: fixing AI crawler access, how to fix robots.txt blocking AI crawlers, and GPTBot and robots.txt.
The free scan gives you the per-bot access matrix, your posture label, a scored verdict on retrieval-bot blocks with the exact robots.txt rule causing each problem. Need a corrected file right now? The free robots.txt generator and llms.txt generator build them. Monitor ($29 a month) rescans weekly, so a deployment that pushes a bad robots.txt rule gets caught. The umbrella view is the AI search optimization hub.
No. GPTBot is OpenAI's training crawler. OAI-SearchBot supports search discovery, while ChatGPT-User performs user-triggered visits. Their behavior and controls differ, so check the operator documentation before changing a rule. Blocking GPTBot does not affect ChatGPT search results. Read more: GPTBot and robots.txt.
Not in traditional search. GPTBot and ClaudeBot training policies are separate from search crawler access. Google-Extended does not affect Google Search inclusion or ranking, but it also controls certain Gemini grounding uses. We say so plainly in the report; we do not pressure you to open training access.
No robots.txt means all crawlers are allowed by default. That is a valid posture. We report it accurately.
Yes. The report names the exact user-agent match and Disallow path that causes each block, so you know precisely what to change.
The robots.txt check applies site-wide. The JavaScript render check runs against both a desktop and mobile fetch of your homepage.
Free scan, no account. Every finding links to the vendor documentation that grounds it, and every verdict is derived from what we actually fetched.
Free category scores and selected findings. Full findings and export files: $10 once.
15 bots by name · retrieval vs training separated · part of the 157-check audit