How do you write a robots.txt file?

What is a robots.txt file?

A robots.txt file is a simple text file that follows the Robots Exclusion Standard. It contains rules that inform search engine crawlers, such as Google's, which URLs they are allowed to access on your site. Before crawling a site, Google's crawlers download and parse this file to understand which parts of the site they may crawl.

The main purpose of a robots.txt file is to prevent your site from being overloaded with requests from crawlers. It is not designed to hide web pages from search results. For the file to work correctly, you must place it in the top-level directory of your site, on a supported protocol. The rules within the robots.txt file apply only to the specific host, protocol, and port number where the file is hosted.

Each line in a robots.txt file consists of a field, followed by a colon, and then a value. For example, a line might look like `User-agent: *` or `Disallow: /private/`. It is important to create this file using a plain text editor, as word processors can introduce proprietary formatting or unexpected characters that may cause issues for crawlers.

How do you write rules in robots.txt?

Robots.txt rules are written using directives that specify a user agent and then either allow or disallow access to certain paths. A valid line always has a field, a colon, and a value. The `User-agent` directive identifies the specific crawler the subsequent rules apply to. For instance, `User-agent: *` applies to all crawlers, except for various AdsBot crawlers, which need to be named explicitly. You can consolidate rules for multiple crawlers into one group, making the file shorter and easier to manage, as all rules in that group will apply to every user agent listed.

The `Allow` and `Disallow` directives control access. For example, `Disallow: /` tells crawlers not to access any part of the site. Conversely, `Allow: /` explicitly permits all crawlers to access the entire site. To disallow crawling of a whole directory, you should append a forward slash to the directory name, such as `Disallow: /private/`.

Google, Bing, and other major search engines support a limited form of wildcards for path values. The asterisk (`*`) can match any sequence of characters. For example, `Disallow: /*.pdf` would block all PDF files. The dollar sign (`$`) wildcard designates the end of the URL. So, `Disallow: /*.pdf$` would block only URLs ending with '.pdf'. When crawlers match robots.txt rules to URLs, they use the most specific rule based on the length of the rule path.

What are common robots.txt rules?

A common rule is to allow all crawlers full access to a site. This is achieved with a `User-agent: *` followed by an `Allow: /` directive. This explicitly allows all crawlers to access the entire site. If you want to block all crawlers from your site, you would use `User-agent: *` followed by `Disallow: /`.

To prevent crawlers from accessing specific directories, you can use the `Disallow` directive with the directory path. For example, `Disallow: /images/` would prevent crawlers from accessing any content within the `/images/` directory. Remember to append a forward slash to the directory name to ensure the rule applies to the entire directory.

Wildcards provide flexibility for more specific blocking. For instance, to block all files with a certain extension, you could use `Disallow: /*.zip$` to prevent crawlers from accessing any ZIP files on your site. Similarly, `Disallow: /temp/*.html` would block all HTML files within a `/temp/` directory, while allowing other file types in that directory.

What can robots.txt not do?

It is important to understand that a robots.txt file is not a mechanism for keeping a web page out of Google Search results, nor is it a way to hide private content. Google documents that it cannot index the content of pages that are disallowed for crawling. However, it may still index the URL and show it in search results without a snippet. This means that even if a page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description.

A page disallowed in robots.txt can still be indexed if it is linked to from other sites. In some situations, URLs from a site may still be indexed even if they have not been crawled. Furthermore, the robots.txt file itself can be viewed by anyone, potentially disclosing the location of your private content. Google also notes that robots.txt rules may not be supported by all search engines.

While Google cannot index images and videos without crawling them, the general principle remains: robots.txt is a crawling directive, not an indexing directive. If you have content you truly want to keep private or out of search results, other methods are more effective.

How do you prevent a page from appearing in search results?

To properly prevent a URL from appearing in Google Search results, you should use more robust methods than robots.txt. One effective approach is to password-protect the files on your server, ensuring that only authorized users can access the content. This is crucial for private content, as robots.txt files are publicly viewable and do not offer true security.

Another method is to use the `noindex` meta tag within the page's HTML or a `noindex` response header. This explicitly tells search engines not to index the page, even if they crawl it. Alternatively, if the page is no longer needed, you can remove it entirely from your server. Google cautions that combining multiple crawling and indexing rules might cause some rules to counteract others, leading to unintended outcomes.

Remember, don't use robots.txt to block access to private content; use proper authentication instead. A robots.txt file can disclose the location of private content, and URLs disallowed by it might still be indexed without being crawled.

How do you test and deploy a robots.txt file?

When creating your robots.txt file, always use a plain text editor. Word processors can save files in proprietary formats or add unexpected characters, such as curly quotes, which can cause problems for crawlers. The file must be located at the root of the site host to which it applies, meaning it should be accessible at a URL like `example.com/robots.txt`.

Once you have created the file, you can upload it to the top-level directory of your site. Google offers tools to help fix issues with robots.txt markup, such as the robots.txt report available in Google Search Console. This report can help you identify and resolve any syntax errors or conflicts in your rules.

After you have uploaded and tested your robots.txt file, Google's crawlers will automatically find and start using it. There is no need to manually submit the file to Google, as crawlers are designed to look for it at the standard root location.

Common questions