AuditLamp Research / Crawler access
Measure the rules.
Keep the evidence.
A dated record of how sampled websites configure crawler access. Each published wave should include its sample, measurements, data, and limits.
Published / Wave 01
Who blocks
the AI crawlers?
The first wave measured robots.txt policy across domains drawn from the Tranco top 1,000. The analysis uses the 653 reachable domains; the full CSV preserves all 1,000 sample rows.
Explore Wave 01 →- Measured
- 5 July 2026
- Published
- 10 July 2026
- Reachable sample
- 653 domains
- Named crawlers
- 15
- Download
- 1,000-row CSV ↓
Reading the series
An access rule
is one piece of evidence.
Training crawlers, AI search crawlers, and traditional search bots can have different rules on the same site. Wave 01 lets you compare the recorded blocking rates for each named crawler.
Those rules do not show whether a bot visited, whether a model used the content, or whether an answer cited the site. The study reports policy measurements without treating them as traffic or citation measurements.
Future waves can support comparisons only when their dates, sample selection, and measurement methods are available. There is one published wave here today; a trend line would imply evidence we do not yet have.
Use the data
Make the source easy to check.
AuditLamp team. Who blocks the AI crawlers? Tranco top-1,000 sample; 653 reachable domains. Measured 5 July 2026; published 10 July 2026. Include the measurement date, sample size, and a link to the full study when citing these results.
Your website
Check your own access rules.
Start with the free scan. Unlock the detailed findings, PDF, action plan, and AI-ready fix file for $10 once.