Fix library

“Blocked by robots.txt” and “Indexed, though blocked by robots.txt”: how to fix both

The AuditLamp team · grounded in the docs · updated 2026-10-18

Both statuses mean a robots.txt rule stops Googlebot from fetching the URL. “Blocked by robots.txt” means Google left the URL out of the index. “Indexed, though blocked by robots.txt” means Google indexed it anyway, from links pointing at it, without ever reading the page. Google explains: “Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page.” The fix depends on what you want: remove the rule if the page should be in Google, or allow crawling and add noindex if it should not.

Decide what the page should do

Start with one question for each URL in the report: should this page appear in Google?

Yes, it should be indexed. The rule is a mistake. Remove or narrow the Disallow that matches it.
No, it should stay out. Robots.txt is the wrong tool. Allow crawling, then add <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header. Google has to fetch the page to see the noindex, so a Disallow actually hides it.
It does not matter. Internal search results, cart steps and filter URLs are often blocked on purpose. If they show as “Blocked by robots.txt” and not indexed, leave them.

How Google decides which rule applies

Most surprises come from how rules are matched, so it helps to know the order. From Google's robots.txt specification and RFC 9309:

One group per crawler. Googlebot obeys only the group that names it, if one exists. Otherwise it obeys User-agent: *. The two groups are never combined, so a Googlebot group with one rule ignores everything under *.
Longest match wins. Within the group, the rule with the longest matching path applies. Allow: /blog/press beats Disallow: /blog for /blog/press-kit.
Allow wins a tie. When an Allow and a Disallow match with the same length, Google uses the less restrictive one.
Wildcards. * matches any run of characters and a trailing $ anchors the end, so Disallow: /*.pdf$ blocks PDFs and nothing else.

# Keep internal search out, let everything else through
User-agent: *
Disallow: /search
Allow: /search-tips

# This group REPLACES the one above for Googlebot
User-agent: Googlebot
Disallow: /drafts/

Check a URL against robots.txt

Paste a URL. We fetch the site's robots.txt from our server and show whether Googlebot, Bingbot and the main AI crawlers may fetch that exact URL. Free, no account, nothing stored.

Causes worth checking first

A staging rule that shipped. User-agent: * plus Disallow: / blocks the whole site. It is the most common cause of a sudden drop in crawling.
A prefix that is too broad. Disallow: /p blocks /pricing and /products too. Rules match from the start of the path.
Blocked CSS or JavaScript. Disallowing /assets/ can stop Google from rendering the page properly, even though the HTML itself is allowed.
A robots.txt that errors. If /robots.txt answers with a 5xx server error, Google stops crawling the site for up to 12 hours, then falls back to its last good copy. A 404 means no rules at all. Make sure the file answers 200 as plain text.

What fixing it will and will not do

Removing a wrong Disallow lets Google fetch the page again. Whether it then indexes or ranks the page is Google's call, based on the page itself. Adding noindex removes a page from the index after the next crawl, which can take days or weeks. Robots.txt is not access control: the file is public, and blocked pages are still reachable by anyone with the URL.

Verify the fix

Run the check above on a URL from each group in the report, and confirm Googlebot shows the result you intended. In Search Console, open Settings, then the robots.txt report, to see the copy Google last fetched and request a recrawl of the file after you change it. Then use URL Inspection on a sample page and Validate fix in the Pages report.