AuditLamp // crawler

The AuditLampAudit crawler

AuditLampAudit is the web crawler AuditLamp uses to audit a website at its owner's request. It identifies itself by name, respects your robots.txt, and rate-limits all requests. The crawler only runs scans initiated by a site owner through AuditLamp. To block it, add User-agent: AuditLampAudit to your robots.txt.

Identifies itself · obeys robots.txt · rate-limits

AuditLampAudit is the crawler AuditLamp uses to read a website and audit it. It runs only when that site's owner asks us to, by submitting the URL. An audit reads a small, bounded set of public pages plus robots.txt, sitemap.xml and llms.txt, and scores what it finds against 164 graded checks: SEO, AEO and GEO plus the Vibe Audit (security, accessibility, privacy).

How it identifies itself

Every request AuditLamp makes carries a user-agent that names us and links back to this page, so you can always tell it apart in your logs:

User-agent (default)Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 AuditLampAudit/1.0 (+https://auditlamp.com/bot)

The browser portion is kept so we can read sites that block unfamiliar clients outright; the AuditLampAudit/1.0 token is the part that identifies us. Operators can switch AuditLamp to a fully transparent, non-browser user-agent if they prefer.

How to allow or block it

AuditLampAudit obeys robots.txt. To control it by name, target the AuditLampAudit token. For example, to block it entirely:

robots.txtUser-agent: AuditLampAudit Disallow: /

We honour Disallow rules for any page we discover while crawling, and we honour a Crawl-delay you set. The one page we always read is the exact URL the owner submits to us for an audit, because that is a direct, owner-initiated request for that page.

How politely it crawls

Questions or problems

If AuditLampAudit is causing trouble, or you want it to stop, email hello@auditlamp.com and we will sort it out quickly.

Policy help: AI bot policy templates · how the scan works.