Robots.txt

Robots.txt is a text file at your domain root that tells compliant crawlers which URLs they can and cannot request. It is a polite request, not a security measure. I have seen people treat it as a way to hide private content, but any bot that does not follow the rules (malicious scrapers, for instance) will ignore it completely. Google respects robots.txt for crawling but explicitly warns that it is not a mechanism for keeping a page out of its index. If a page blocked by robots.txt has enough external links, Google may still index it based on those signals without ever crawling the page.

The most practical use is controlling crawl budget. For a site with 10,000 product pages and 50,000 filter-parameter URLs, a thoughtful robots.txt can reduce weekly crawl requests from 200,000 to 40,000, shifting effort to pages that convert. I always start by blocking known waste: admin paths, login screens, search results, and paginated archives that are canonicalised elsewhere. Some SEOs argue you should never block anything because Google might need it. I disagree. The risk of blocking something valuable is real, but the risk of wasting crawl budget on infinite parameter combinations is often larger. The key is testing: use the robots.txt tester in Search Console before going live.

Robots.txt has the most impact on large sites or those with dynamic URL structures. If your site has fewer than 1,000 URLs, Google will likely crawl everything it finds without any rules. The file still matters for preventing access to non-public paths, but the crawl-efficiency gain is negligible. For larger sites, the difference can be dramatic.

Consider an e-commerce site with faceted navigation. Colour, size, price-range filters can generate hundreds of thousands of parameterised URLs. Without robots.txt, Googlebot may spend 70% of its crawl budget on these near-duplicate filter pages. I worked on a site where blocking the filter parameter path reduced crawl requests from 150,000 to 30,000 per week, and the product pages started being re-crawled twice as often. That directly improved indexing speed for new inventory.

Robots.txt also matters when you have a subdomain or separate directory for staging content that should never be crawled. I always add a Disallow for the entire staging environment. However, this only works if the staging site lives on a separate subdomain. If it is in a subdirectory of the main domain, you cannot reliably block it from authenticated users, but you can prevent Google from crawling it.

An edge case: if you block your JavaScript or CSS files, Google cannot render your pages properly. I have seen a 40% drop in featured snippet appearances after a team accidentally blocked a /js/ directory. Always allow the assets needed for rendering.

The robots.txt file lives at the root of your host, like https://example.com/robots.txt. It uses a simple syntax: User-agent: to specify the crawler, then Disallow: lines to block paths, and optionally Allow: to override a broader block. The rules are evaluated top to bottom per user-agent group. The most specific user-agent wins: a rule for Googlebot overrides a rule for *.

I always include a Sitemap: directive pointing to my main [sitemap](/sitemap/) location. This helps search engines discover new pages even if the rest of the file is restrictive. For example, if I block /blog/ but want the blog sitemap crawled, I place the sitemap outside the blocked directory or add an Allow for it.

You can see robots.txt in action inside Search Console's Crawl Stats report. If you block a section, you should see a sharp drop in crawl requests for those URLs. If you do not see a change, either the rules are wrong or Google has not re-fetched the file. The file is fetched periodically and cached.

One detail many miss: the file must be publicly accessible without authentication. If you require login, Google cannot read it. Also, the file size limit is 500 KiB for Google, but I keep mine under 50 KiB. Large files risk Google ignoring the later lines.

The primary tradeoff is between crawl efficiency and risking the blocking of important resources. A too-aggressive Disallow on a directory that contains critical CSS or JS files can break Google's ability to render pages. I learned this when a client blocked /assets/ and their [Core Web Vitals](/core-web-vitals/) scores plummeted because Google could not see the rendered page.

Another tradeoff: using robots.txt to block [duplicate content](/duplicate-content/) is tempting but ineffective. Google can still index blocked URLs if external sites link to them, and then you have indexed pages you cannot easily noindex because the page itself is not crawled to see the noindex tag. That is a double bind. The correct approach is to use [canonical tags](/canonical-tag/) or the noindex directive (combined with a clean crawl path). For pages you do not want crawled at all, robots.txt is appropriate; for pages you do not want indexed, use indexing controls and ensure they are not blocked from crawling.

A third tradeoff involves sitemap discovery. If you block a directory that contains your sitemap, Google cannot fetch it. I always place sitemaps outside any blocked directories or add explicit Allow directives for them.

There is also a performance cost: each disallowed path adds overhead to Google's parsing of the file. I limit the number of rules to under 50. For large sites with many parameters, I use pattern-based blocking (like Disallow: /?filter=) instead of listing every parameter.

Decision rule: I block only what I am certain has zero SEO value and cannot be fixed with better canonicalisation. I never block CSS, JS, or images from main content directories.

I used robots.txt to block thin content pages expecting them to drop from the index. They did not. Google kept them indexed because other pages linked to them, and since Google could not crawl them, they could not see the noindex tag I later added. The lesson: robots.txt controls crawling, not indexing. If you want a page out of the index, use a noindex tag and leave it crawlable.

I also once accidentally blocked an entire subdomain. I wrote Disallow: /blog/ intending to block just the blog directory, but I had the sitemap directive pointing to /blog/sitemap.xml. Google could not crawl the sitemap because it was inside the blocked directory. I lost 30% of indexed blog posts before I spotted the problem in the robots.txt tester.

Another mistake: I blocked all bots except Googlebot on a site hoping to save server resources. I did not realise that Bing was sending decent referral traffic. It took me a month to notice the dip in organic visitors from Bing. Now I only block specific user-agents if they are abusive, not as a broad policy.

I also used to think robots.txt could stop competitors from [scraping my content](/web-scraping/). It cannot. If you need to protect content, use authentication or legal measures. Robots.txt is an opt-out for well-behaved bots only.