Robots.txt

Robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages or sections they are allowed or disallowed from crawling. It is used to manage crawl budget and prevent indexing of duplicate, private, or low-value pages. Misconfiguring robots.txt can accidentally block important content from appearing in search results. Why it matters: robots.txt is one of the most powerful yet frequently misunderstood tools in technical SEO. A single misplaced directive can prevent entire sections of your site from being crawled and indexed, effectively making them invisible to search engines. Common use cases include blocking admin directories, staging environments, internal search result pages, and faceted navigation URLs that create duplicate content. However, it is important to understand that robots.txt prevents crawling, not indexing. If other sites link to a disallowed page, Google may still index the URL based on anchor text alone without crawling its content. To prevent indexing entirely, a noindex meta tag or X-Robots-Tag header is the correct approach. Best practices include testing your robots.txt file with the Google Search Console robots.txt tester before deploying changes, referencing your XML sitemap location within the file, and using specific user-agent directives only when necessary. For most sites, a simple robots.txt that allows all important content, blocks only administrative and duplicate paths, and points to the sitemap provides the right balance of crawl efficiency and content accessibility.