What is the meta robots tag, and which values matter?
guide // , 5 min read, by Darthwares
The meta robots tag and the X-Robots-Tag HTTP header are tools that give you granular control over how search engines interact with your website's content. These directives tell crawlers whether to index a page, follow its links, or how to display it in search results. Understanding how to use them correctly is essential for managing your site's visibility in search engines.
On this page // 6 sections
The short version
- The meta robots tag controls indexing for HTML pages, while the X-Robots-Tag header is for non-HTML resources like PDFs.
- Both methods require crawlers to access the page or resource to read the directives.
- The `noindex` directive prevents a page from appearing in search results, but it only works if the page is crawlable.
- The `nofollow` directive tells crawlers not to follow links on a page.
- When rules conflict, the more restrictive rule always applies.
What is the meta robots tag?
The meta robots tag is a piece of code placed within the HTML of a web page that provides instructions to search engine crawlers. Google documents that this tag allows for a page-specific approach to controlling how an individual HTML page should be indexed and served in Google Search results. For these settings to be effective, however, crawlers must be able to access the pages that contain these meta tags.
What is the X-Robots-Tag HTTP header?
The X-Robots-Tag is an HTTP response header that defines how crawlers should index URLs. MDN Web Docs states that while it is not part of any official specification, it has become a de-facto standard for communicating with search bots and web crawlers. This method is especially useful for non-HTML documents, such as PDF files, video files, or image files, where a meta tag cannot be embedded.
Similar to the meta robots tag, the X-Robots-Tag header's rules are only followed by cooperative robots. A crawler must first be able to access the resource to read its HTTP headers and apply the specified indexing rules.
How do meta robots tags and X-Robots-Tags interact?
Both the meta robots tag and the X-Robots-Tag HTTP header provide instructions to search engine crawlers. A critical point for both is that the page or resource must be accessible to the crawler for the rules to be discovered and applied. If a page's path is blocked from being crawled by a robots.txt file, for example, the crawler will not be able to read the meta tag or HTTP header, and thus cannot apply the indexing rules.
In situations where there are conflicting robots rules, whether within an X-Robots-Tag, or between an X-Robots-Tag HTTP header and the meta robots tag, the more restrictive rule always applies. This ensures that if you have multiple directives, the one that limits indexing or crawling the most will take precedence.
What does noindex mean?
The `noindex` directive instructs search engines not to show a specific page, media, or resource in search results. Google documents that when Googlebot crawls a page and finds a `noindex` tag or header, Google will drop that page entirely from its Search results, even if other websites link to it. Google also notes that using `noindex` is useful if you don't have root access to your server, as it allows you to control access to your site on a page-by-page basis.
For the `noindex` rule to be effective, the page or resource must not be blocked by a robots.txt file and must be otherwise accessible to the crawler. If a page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the `noindex` rule, and the page can still appear in search results, for example if other pages link to it. Google does not support specifying the `noindex` rule within a robots.txt file. You can request that Google recrawl a page using the URL Inspection tool, and to test if your `noindex` implementation is correct, use the URL Inspection tool to see the HTML that Googlebot received while crawling the page.
What do nofollow and max-image-preview mean?
The `nofollow` directive tells search engine crawlers not to follow any of the links on the page where it is present. This rule can be applied using both the meta robots tag and the X-Robots-Tag HTTP header. Google also documents the `max-image-preview` directive, which allows you to set the maximum size of an image preview for this page in search results, giving you control over how your images appear in snippets.
Common questions
Can I use noindex in robots.txt?
No, Google does not support specifying the `noindex` rule in a robots.txt file. For the `noindex` directive to be effective, it must be placed in a meta tag on an HTML page or in an X-Robots-Tag HTTP header, and the page must be crawlable.
What happens if a page with noindex is blocked by robots.txt?
If a page containing a `noindex` rule is blocked by a robots.txt file, the crawler will never be able to access the page to read the `noindex` directive. In this scenario, the page can still appear in search results, especially if other pages link to it, because the `noindex` instruction was never discovered.
How can I check if my noindex rule is working correctly?
You can test your `noindex` implementation using Google's URL Inspection tool. This tool allows you to see the HTML that Googlebot received when it crawled the page, confirming whether the `noindex` tag or X-Robots-Tag header was successfully discovered and applied.
When should I use the X-Robots-Tag instead of the meta robots tag?
You should use the X-Robots-Tag HTTP header for non-HTML resources, such as PDF files, video files, or image files. The meta robots tag is embedded within the HTML of a page, making it suitable only for HTML documents. The X-Robots-Tag allows you to apply indexing rules to other types of web content.
What if I have conflicting robots rules?
In cases where there are conflicting robots rules, such as between an X-Robots-Tag HTTP header and a meta robots tag, the more restrictive rule always applies. This principle ensures that if you have multiple directives, the one that limits indexing or crawling the most will take precedence.
Primary sources and further reading
This article was researched and written by an AI model and published automatically. It was written from sentences that were checked to appear on the pages below, but a person has not reviewed the text. If something is wrong, tell us at chibivikramathithan@darthwares.com and we will correct it.