Block training. Keep answers. Or the reverse. Just decide on purpose.
AI crawlers are not one switch. Training bots and search or answer bots are often different names with different jobs. Most sites inherit a CDN default and never notice. This page helps you pick a posture. The free scan shows whether your live site matches it.
Whole-site scan · named bots · no email on the diagnosis
Three jobs, not one bot.
Training
Bots that help build model weights (examples operators document: GPTBot, ClaudeBot, Google-Extended). Blocking them is a content-use policy choice. It is not the same as leaving AI search.
Search and answers
Bots that help answer engines and AI search index or fetch pages for citations (examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking these can hide you from people who ask AI who to hire or what to buy.
User-triggered fetch
Some agents fetch a page because a human asked about that URL (example: ChatGPT-User). Different from bulk training. Still a machine reading your public HTML.
Our scan reports a per-bot matrix and live reachability probes. Named checks include gate.ai_bot_access, gate.bot_reachability, and core.train_vs_retrieve. Robots can say allow while the edge still blocks: robots say yes, CDN may still block.
Pick the business you run. Then verify live.
A. Lead-gen SMB / local service (plumber, clinic, firm, restaurant) Goal: show up when people ask AI for recommendations. Posture: allow search and answer bots; training is optional (many SMBs leave training allowed by default). First fix if blocked: clear edge and robots blocks on search bots, then re-scan.
B. Publisher / media Goal: often allow citation and search while limiting training. Posture: allow search/answer bots; consider blocking training bots if that is legal/policy intent. First fix: split rules by bot name, not a blanket "Disallow: /" for all AI UAs.
C. SaaS docs / product site Goal: docs and comparison pages stay machine-readable for shortlists. Posture: allow search/answer bots on public marketing and docs; keep app routes out of sitemap and robots as you already would for private app. First fix: no JS-only docs shell; bots must see text without scripts.
These are templates, not legal advice. Operators change bot names and roles. Always check the vendor's current documentation, then measure your live site.
Honest limits.
We do not promise that allowing any bot will produce citations or traffic. Models personalize and change. We measure access and readiness on your public pages.
We do not replace your lawyer or compliance team for training opt-outs. We show the technical posture.
Google Search and AI Overviews are not controlled by Google-Extended alone. Do not treat one token as a magic AIO switch. More in how it works and readiness vs monitoring.
Measure, then edit robots or CDN.
- Run the free scan and open the AI bot / access findings.
- Compare the matrix to template A, B, or C above.
- Change robots.txt and CDN bot rules so they match. Free helper: AI crawler access checker and robots.txt generator.
- Re-scan. If you want ongoing watch after deploys, that is Monitor $29.
Deeper notes: GPTBot blocked, what we check · crawler blocking study.