AuditLamp Research / Wave 01
Who blocks
the AI crawlers?
Sites can set different rules for AI training, AI search, and traditional search. We measured those differences across a sample from the Tranco top 1,000.
The sample, before the headline
- Domains selected
- 1,000
- Reachable domains
- 653
- Not reached; excluded from rates
- 347
- Named crawlers evaluated
- 15
Results describe this dated sample. They do not estimate the whole web.
01 / The clearest comparison
One company.
Two different access patterns.
OpenAI's GPTBot was blocked by 109 of the 653 reachable sites. Its search crawler, OAI-SearchBot, was blocked by 60. The recorded rules distinguish the two; the data does not tell us why each publisher chose them.
Both bars use the same 0–20% scale. Percentages are rounded to one decimal place.
02 / Explore the results
Every crawler.
The same denominator.
Compare the share of reachable sites whose recorded robots.txt rules blocked each crawler. Filtering changes the rows shown, never the scale or sample.
A robots.txt rule is an access instruction. These measurements do not establish whether a crawler actually visited, whether a model used the content, or whether a site was cited in an answer.
03 / Method and limits
What was measured.
What remains unknown.
A result is only useful if its boundary is clear. The complete CSV includes unsuccessful requests so the exclusions remain visible.
How to reproduce the figures
- Start with the 1,000 domain rows in the downloadable CSV.
- Keep the 653 rows where reachable = 1. The remaining 347 are excluded from these rates.
- For each named crawler, count rows whose corresponding block_… field equals 1.
- Divide that count by 653, multiply by 100, and round to one decimal place.
How to interpret them
The sample is drawn from highly ranked domains and has a substantial unreachable group. It is not a representative estimate for small business websites or the web as a whole.
Recorded rules may change after the measurement date. A permitted user-agent can still encounter a firewall, authentication requirement, or other access restriction.
Training, retrieval, and search labels help explain the comparison. Allowing a crawler does not guarantee indexing, rankings, training use, or AI citations.
04 / Use the evidence
Download it.
Check it. Cite it.
AuditLamp team. Who blocks the AI crawlers? Robots.txt policy in the Tranco top 1,000; 653 reachable domains. Measured 5 July 2026. Published 10 July 2026.
Free to use with attribution. Link to this study and include the measurement date and sample size.
Your website
Check your own crawler access.
See category scores and selected findings free. A Full Report unlocks the detailed findings and files for $10 once.