GPTBot is now the most blocked crawler on the web. Your site might be blocking it without you knowing.
If you want to show up in AI answers, there is a quiet way to lose before you start: block the crawlers that feed those answers. A report making the rounds this week says that is happening at scale.
Per sitestatsdb, shared on Hacker News on July 7, 2026, GPTBot, OpenAI's crawler, is the most blocked crawler among websites that get organic search traffic (sitestatsdb.com). We have not independently reproduced the underlying counts, so we will not put a number on it here. We are pointing at the finding because the direction matters more than the decimal: a lot of sites that clearly care about being found are, at the same time, closing the door on the tools that increasingly decide what gets found.
Most of the time, nobody chose this
Here is the part that catches non-technical owners. Blocking an AI crawler is rarely a decision someone sat down and made. It shows up by accident, through a handful of ordinary paths:
- A security plugin or CDN setting flips on "block AI bots" as a default, and the box stays checked.
- A robots.txt file gets copied from a template or a forum post that already had `Disallow` rules for GPTBot, CCBot, ClaudeBot, or Google-Extended.
- A well-meaning developer adds a rule during a content-scraping scare and never removes it.
- A hosting provider ships an AI-blocking option in a dashboard the owner has never opened.
None of these are wrong on their own. The problem is that the outcome, being invisible to AI answer engines, is a big deal, and it was reached without anyone weighing it.
Two different goals, two different controls
There is real nuance here, and we are not going to flatten it. Blocking AI crawlers can be a completely reasonable choice. Some owners do not want their writing used to train models, and that is their call to make.
But two goals often get tangled together:
- Do not let my content be scraped for AI training.
- Do show up when someone asks an AI assistant a question my business could answer.
These pull in different directions, and the controls are not the same. A blanket block aimed at goal one can quietly cost you goal two. If you have never separated the two in your own settings, the odds are good that a default separated them for you, and not in the direction you would have picked.
What to actually check
You do not need to guess. You can read your own site the way a crawler does. A few concrete things to look at:
- Your `robots.txt`. Look for `Disallow` lines tied to user agents like `GPTBot`, `Google-Extended`, `ClaudeBot`, `PerplexityBot`, `CCBot`, or a catch-all `*` that sweeps them in.
- Your CDN or firewall bot rules. Cloudflare, and similar services, expose an "AI bots" toggle that can block at the edge before your site is ever consulted.
- Whether your content is even reachable without JavaScript. A crawler that gets an empty shell sees nothing to quote, which is its own kind of block.
Decide each of these on purpose. Keep the doors you mean to keep closed closed, and open the ones you did not mean to shut.
Our take
We build AuditLamp to tell owners the plain truth about what search engines and AI answer engines actually see when they read a site. Whether you are blocking the crawlers you want to reach you is one of the first questions that truth answers, and a surprising number of owners are getting a different answer than they expect.
The free scan reads your site the way these crawlers do and tells you, in plain language, whether you are open or closed to them, and where the block is set if there is one. From there, the choice is yours to make on purpose. No email required to get your diagnosis.
Sources: sitestatsdb, "GPTBot is the most blocked crawler among websites that get organic traffic," shared on Hacker News, July 7, 2026 (sitestatsdb.com).