ChatGPT, Claude and Perplexity each send a named crawler to read your pages. One line in your robots.txt, or one firewall rule, and that engine never sees you, so it can never recommend you. We read your robots.txt the way those crawlers do and tell you exactly who gets in. No email.
free / no email / ~30s
crawler access is 1 gate of the full 116-check audit
Every AI vendor documents its crawlers by name, and they are independent controls. We test each one against your live robots.txt using the vendors' own documented user-agent tokens, plus the wildcard and longest-match rules robots.txt actually uses.
Allow the search-class crawlers if you want customers to find you through AI: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude search) and PerplexityBot (Perplexity). Those are the ones that decide whether an engine can read and cite your pages in its answers. The training crawlers, GPTBot, ClaudeBot, Google-Extended and CCBot, only feed model training, so allowing or blocking them is a business decision about your content, not a visibility decision.
No. GPTBot is OpenAI's training crawler; blocking it opts your content out of model training and nothing else. ChatGPT's search results come from a different crawler, OAI-SearchBot, which OpenAI documents as an independent control. The same split exists at Anthropic (ClaudeBot for training, Claude-SearchBot for search). Many sites copy a blanket block-everything robots.txt from a blog post and accidentally block the search crawlers too, which is the mistake this checker catches.
No. Google's AI-search guidance says outright that you do not need new machine-readable files or AI text files to appear in generative AI search. An llms.txt is an optional, unofficial experiment: a short index of your key pages that a few non-Google AI tools read. It costs almost nothing to publish, so we report whether you have one, but no engine vendor requires it and skipping it does not hold you back with Google.
Getting the crawlers through the door only matters if what they find is worth citing. The full AuditLamp audit runs 116 checks across SEO, answer-box eligibility and AI-assistant readiness on the same free scan: full score, every gate failure, your worst problems, on screen, no address taken. We make money on the watching and the fixes, not your inbox.