robots.txt and meta robots tags, explained

robots.txt sits at the root of a domain, for example example.com/robots.txt, and gives instructions to crawlers before they request anything else on the site. It is a crawling directive, not a security mechanism and not an indexing mechanism.

Two things follow from that. First, it is a suggestion. Google, Bing and other major search engines respect it, but plenty of scrapers and bad actors ignore it entirely, so anything genuinely sensitive, a customer database export, an internal API key, an unlaunched product page, needs to sit behind authentication, not behind a Disallow line. robots.txt is public: anyone can read it at yourdomain.com/robots.txt, which means it can also hand an attacker a map of the folders you'd rather they not look at.

Second, disallowing a URL in robots.txt does not guarantee it stays out of Google's index. If other sites link to a blocked URL, Google can still show it in results, usually as a bare URL with no title or description, because it was never allowed to crawl the page to read a noindex tag or generate a snippet. This is the single most common misunderstanding about the file.

The file is plain text, one directive per line, grouped under a User-agent line. A group applies to whichever crawler matches that user-agent, and an asterisk matches all of them.

The core directives are simple, but the matching rules trip people up. Disallow and Allow match by path prefix, and the most specific matching rule wins, not the first one listed. A trailing slash matters: Disallow: /search blocks /search and /search-results, while Disallow: /search/ only blocks paths under that folder.

This is roughly what a small business site on WordPress or a similar CMS should ship. Adjust the folder names to match your platform, but the shape holds.

Use a meta robots noindex tag, placed in the page's head as <meta name="robots" content="noindex">, when you want a page kept out of search results but you still want it crawled. Crawling has to be allowed, because Google can only obey a noindex tag if it is permitted to fetch the page and read it.

This covers most of the pages people reach for robots.txt first: thank-you pages after a form submission, internal search result pages, thin tag or author archive pages, staging content that accidentally shipped to production, duplicate print or PDF versions of a page that already exists as HTML. In each case the page needs to exist and be reachable, it just should not compete for a place in results.

The reverse combination, noindex plus Disallow on the same URL, quietly breaks things. Google cannot recrawl a blocked page to notice the noindex tag went away, and it cannot always drop an already-indexed page from results if it is blocked from rechecking it. If a page must come out of the index, allow crawling, add noindex, wait for Google to process it via Search Console's URL Inspection tool, then block it in robots.txt afterward if you want to save crawl budget going forward.

X-Robots-Tag is the HTTP-header equivalent of meta robots, useful for non-HTML files like PDFs where you cannot inject a meta tag. It goes in the server response header and works the same way: it needs crawling allowed to take effect.

None of these need to be indexed and most add nothing to what a crawler already understands about your site, so keeping them out of the crawl queue is pure upside for crawl efficiency.

The failures here are rarer than over-blocking low-value pages, but they are far more damaging because they can silently remove an entire site from search.