robots.txt is the first file every crawler reads. One wrong line and you can wave off Google, or the AI engines, without meaning to. This builds a valid file in your browser: choose what to allow, disallow the paths that should stay private, point to your sitemap, and allow or block named AI crawlers on purpose. One honest thing up front, from Google's own docs: robots.txt controls crawling, not indexing. A blocked URL can still show up in results. To keep a page out of the index, use a noindex meta tag, not a Disallow.
yourdomain.com/robots.txt. It builds here as you type and never leaves your browser.robots.txt is 1 of the 116 checks in the full auditlamp audit
The most expensive robots.txt mistakes come from believing the file does more than it does. It is a set of polite requests that well-behaved crawlers follow. It is not a lock, not a firewall, and not a way to remove a page from Google. Here is what the lines actually mean.
Disallow stops Google from fetching the page, but if other pages link to it, the URL can still appear in results, usually without a description. To keep a page out of the index, let it be crawled and add a <meta name="robots" content="noindex"> tag, or return an X-Robots-Tag: noindex header. A Disallowed page can never be seen, so its noindex is never read.Allow: /blog/public/ beats Disallow: /blog/ for URLs under that folder. Keep paths simple and test them; a stray slash changes the scope./Admin/ and /admin/ are different rules. Paths are relative to the root and begin with /. This generator adds the leading slash for you and leaves your casing alone.OAI-SearchBot or PerplexityBot can make you invisible in ChatGPT and Perplexity answers. Blocking GPTBot or ClaudeBot only opts your content out of model training and does not affect those engines' search answers. They are independent tokens, documented by each vendor, which is why we group them by what blocking them costs you.Sitemap: line points crawlers at your full URL list. It must be a complete address starting with https, not a relative path. It is the one line that helps discovery instead of restricting it.This generator writes standard directives that Google, Bing and the documented AI crawlers understand. It does not emit Crawl-delay, because Google ignores it. When in doubt, keep the file short: the fewer rules you write, the fewer ways there are to accidentally block yourself.
No. robots.txt controls whether a crawler is allowed to fetch a page, not whether the page can be indexed. Google states plainly that a URL blocked in robots.txt can still appear in search results if other pages link to it, just without a snippet, because Google never fetched the content. If your goal is to keep a page out of the index, do the opposite of blocking it: allow crawling and add a noindex meta tag or X-Robots-Tag header, so the crawler can actually read the instruction to drop it.
That is a content decision, not a visibility one. GPTBot and ClaudeBot are training crawlers: blocking them opts your pages out of model training and nothing else. It does not remove you from ChatGPT or Claude search answers, which come from different crawlers (OAI-SearchBot and Claude-SearchBot). If you want customers to keep finding you through AI, leave the search-class crawlers allowed; block the training crawlers only if you have a specific reason to withhold your content from model training.
It must live at the root of your domain, so it loads at yourdomain.com/robots.txt, served as plain text. A crawler only looks in that one spot; a robots.txt in a subfolder is ignored. Each subdomain needs its own file, so blog.yourdomain.com and yourdomain.com are controlled separately. After you upload it, open the address in a browser: if you can read the file there, so can every crawler that checks it.
Getting crawlers through the door only matters if what they find is worth indexing and citing. The full AuditLamp audit runs 116 documented checks on your live site in one free scan: whether crawlers can reach you, whether anything blocks your snippets, whether your content survives without JavaScript, and whether there is an answer worth quoting near the top. Full score on screen, no email taken.