How many AI crawlers exist, and what do they do?
AuditLamp's engine currently tracks 15 AI crawlers by user-agent string, drawn from their vendors' official documentation. They split into two categories with very different implications: 6 retrieval crawlers that fetch pages to build live AI answers (blocking these removes you from AI search results), and 9 training crawlers that scrape content to train AI models (blocking these is a content licensing choice, not an AI-search choice). AuditLamp engine roster, OpenAI bot docs; Anthropic bot docs; vendor docs for each bot listed below.
What is the difference between a retrieval crawler and a training crawler?
A retrieval crawler fetches your pages in real time when a user asks an AI search engine a question. If you block a retrieval crawler, your site does not appear in that engine's answers. A training crawler scrapes your pages to build the language model itself. Blocking a training crawler keeps your content out of future model training; it has no effect on whether the engine cites you in live answers.
This distinction matters because most discussion about "blocking AI crawlers" conflates the two. Blocking GPTBot (training) is a content licensing decision. Blocking OAI-SearchBot (retrieval) is an AI search visibility decision. They are different user-agents, different robots.txt entries, and different consequences.
Which AI crawlers affect whether you show up in ChatGPT or Perplexity?
The 6 retrieval crawlers in our roster are the ones that feed live AI answers. Run a live check to see which AI bots can read your pages right now.
| Crawler | Vendor | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Fetches pages for ChatGPT web search answers |
| ChatGPT-User | OpenAI | User-initiated browsing in ChatGPT |
| Claude-User | Anthropic | User-initiated browsing in Claude |
| Claude-SearchBot | Anthropic | Fetches pages for Claude search answers |
| Perplexity-User | Perplexity | Fetches pages for Perplexity answers |
| DuckAssistBot | DuckDuckGo | Fetches sources for DuckAssist AI answers |
Sources: platform.openai.com/docs/bots, support.anthropic.com, docs.perplexity.ai/guides/bots.
Which AI crawlers are blocked most often, and why does it matter?
Training crawlers are blocked far more than retrieval crawlers -- a pattern consistent across every dataset we have measured. From AuditLamp's Tranco top-1,000 scan (653 reachable sites, 2026-07-05): GPTBot blocked by 16.7% of sites; OAI-SearchBot blocked by 9.2%; CCBot by 18.7%; Googlebot by 2.0%. See which crawlers are blocked most often by real sites for the full breakdown by crawler.
The gap matters because it shows most blocking is a decision about training data, not about search visibility. A site that blocks GPTBot but allows OAI-SearchBot is making exactly the right separation: keep content out of model training, stay visible in ChatGPT answers. A site that blocks both has made a different choice. Most sites have not thought about the distinction at all. How to configure GPTBot and AI crawlers in your robots.txt covers the exact syntax for each decision.
How do you check which AI crawlers your site allows?
Two places to check: your robots.txt file (states your policy) and your server's HTTP response to each crawler user-agent (measures the actual result). They can disagree. In our scan of 102 sites, 12.7% blocked AI crawlers at the CDN level even though their robots.txt said the crawlers were welcome.
The 9 training crawlers in our roster are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Googlebot (Google -- included for full-roster context as it is tracked in our engine), AppleBot (Apple), FacebookExternalHit (Meta), CCBot (Common Crawl), Amazonbot (Amazon), and Bytespider (ByteDance). Our roster is sourced from vendor documentation and may not be exhaustive; new AI crawlers launch regularly. Check your full site's AI crawler access settings to see what each of these bots currently gets from your server. Check which of these AI crawlers can access your site as part of the full AEO audit.
Cite this list
AuditLamp. "How many AI crawlers exist, and what do they do?" AuditLamp engine bot registry, sourced from vendor documentation, updated 2026-08-25. auditlamp.com/learn/notes/how-many-ai-crawlers-exist. Free to use with attribution; verify vendor docs for the latest user-agent strings.