auditlamp tools robots.txt // free

Free robots.txt generator for AI crawlers

robots.txt is the first file every crawler reads. One wrong line and you can wave off Google, or the AI engines, without meaning to. This builds a valid file in your browser: choose what to allow, disallow the paths that should stay private, point to your sitemap, and allow or block named AI crawlers on purpose. One honest thing up front, from Google's own docs: robots.txt controls crawling, not indexing. A blocked URL can still show up in results. To keep a page out of the index, use a noindex meta tag, not a Disallow.

search crawlers · blocking hides you from that engine's answers
training crawlers · blocking is a legitimate opt-out
robots.txt

        
Upload this to the root of your site so it loads at yourdomain.com/robots.txt. It builds here as you type and never leaves your browser.

      

    

    
robots.txt is 1 of the 164 checks in the full auditlamp audit

  






  

    
what a robots.txt actually does

    
It tells crawlers where to go. It does not hide you.

    
The most expensive robots.txt mistakes come from believing the file does more than it does. It is a set of polite requests that well-behaved crawlers follow. It is not a lock, not a firewall, and not a way to remove a page from Google. Here is what the lines actually mean.

    
    
      
  • Disallow blocks crawling, not indexingGoogle is explicit: a Disallow stops Google from fetching the page, but if other pages link to it, the URL can still appear in results, usually without a description. To keep a page out of the index, let it be crawled and add a <meta name="robots" content="noindex"> tag, or return an X-Robots-Tag: noindex header. A Disallowed page can never be seen, so its noindex is never read.
    • 
      
  • Allow and Disallow match by longest, most specific ruleCrawlers do not read top to bottom and stop. Google follows the most specific matching path, so Allow: /blog/public/ beats Disallow: /blog/ for URLs under that folder. Keep paths simple and test them; a stray slash changes the scope.
    • 
      
  • Every path is case sensitive and starts with a slash/Admin/ and /admin/ are different rules. Paths are relative to the root and begin with /. This generator adds the leading slash for you and leaves your casing alone.
    • 
      
  • AI crawlers are separate, named controlsBlocking OAI-SearchBot or PerplexityBot can make you invisible in ChatGPT and Perplexity answers. Blocking GPTBot or ClaudeBot only opts your content out of model training and does not affect those engines' search answers. They are independent tokens, documented by each vendor, which is why we group them by what blocking them costs you.
    • 
      
  • Sitemap belongs here, and it is absoluteA Sitemap: line points crawlers at your full URL list. It must be a complete address starting with https, not a relative path. It is the one line that helps discovery instead of restricting it.
    • 
    

    
This generator writes standard directives that Google, Bing and the documented AI crawlers understand. It does not emit Crawl-delay, because Google ignores it. When in doubt, keep the file short: the fewer rules you write, the fewer ways there are to accidentally block yourself.

    
If a rule in your current file is already blocking a crawler you want, our guide to fixing robots.txt rules that block AI crawlers shows the repair, and how to fix canonical tag errors covers the other quiet way ranking signals end up at the wrong URL.

    
After writing your allow rules, check whether they work in practice. The AI crawler access checker probes your live site with real bot user-agents and reports which crawlers get through and which are turned away.

    
Your robots.txt controls which AI crawlers can reach your pages -- run a full GEO audit to see exactly which bots are blocked and what content they cannot see.

    
After generating the file, verify it actually allows the bots you want before deploying it -- run the generated rules through the free AI crawler access checker.

  






  

    
robots.txt questions, answered straight

    
The honest version.

    
Does robots.txt stop a page from showing up in Google?

    
No. robots.txt controls whether a crawler is allowed to fetch a page, not whether the page can be indexed. Google states plainly that a URL blocked in robots.txt can still appear in search results if other pages link to it, just without a snippet, because Google never fetched the content. If your goal is to keep a page out of the index, do the opposite of blocking it: allow crawling and add a noindex meta tag or X-Robots-Tag header, so the crawler can actually read the instruction to drop it.

    
Should I block GPTBot and ClaudeBot in my robots.txt?

    
That is a content decision, not a visibility one. GPTBot and ClaudeBot are training crawlers: blocking them opts your pages out of model training and nothing else. It does not remove you from ChatGPT or Claude search answers, which come from different crawlers (OAI-SearchBot and Claude-SearchBot). If you want customers to keep finding you through AI, leave the search-class crawlers allowed; block the training crawlers only if you have a specific reason to withhold your content from model training.

    
Where does the robots.txt file go?

    
It must live at the root of your domain, so it loads at yourdomain.com/robots.txt, served as plain text. A crawler only looks in that one spot; a robots.txt in a subfolder is ignored. Each subdomain needs its own file, so blog.yourdomain.com and yourdomain.com are controlled separately. After you upload it, open the address in a browser: if you can read the file there, so can every crawler that checks it.

  






  

    
one file of the whole picture

    
A clean robots.txt is one of 164 checks. We run them all.

    
Getting crawlers through the door only matters if what they find is worth indexing and citing. The full AuditLamp audit runs 164 documented checks on your live site in one free scan: whether crawlers can reach you, whether anything blocks your snippets, whether your content survives without JavaScript, and whether there is an answer worth quoting near the top. Full score on screen, no email taken.