What is Google-Extended, and should you block it?

What is Google-Extended?

Google-Extended is a user-agent token that Google's crawlers and fetchers use to identify themselves. Google documents that it uses various crawlers and fetchers to perform actions for its many products, either automatically or when a user triggers a request. A crawler is a program that automatically discovers and scans websites, while a fetcher typically makes a single request on behalf of a user, similar to a program like wget. Googlebot is a common crawler used for Google's products.

These Google crawlers identify themselves in several ways, including through the HTTP user-agent request header, the source IP address of the request, and the reverse DNS hostname of the source IP. In the context of robots.txt files, crawlers use a product token to match and obey specific rules. Google-Extended is an example of such a product token. This allows website owners to manage how content is found and parsed for display in Google Search and other Google properties.

How do you control Google-Extended?

You can control which files Google-Extended, and other crawlers, may access on your site by creating a robots.txt file. Google for Developers states that this file is a plain text document that follows the Robots Exclusion Standard and must be located at the root of your website, for example, at example.com/robots.txt. The IETF's Robots Exclusion Protocol specifies that the file must be UTF-8 encoded, which includes ASCII.

Within the robots.txt file, you define rules that either block or allow access for specific crawlers, or all crawlers, to specified file paths on your domain or subdomain. By default, all files on your site are implicitly allowed for crawling unless you explicitly specify otherwise in your robots.txt file. Google's crawlers always respect these robots.txt rules for automatic crawls.

When a crawler, such as one identified by the Google-Extended token, evaluates access to a page, it uses case-insensitive matching to find the group of rules that correspond to its product token. It then matches the paths in "allow" and "disallow" rules against the page's URL. The most specific match found for that crawler determines whether access is permitted. You can target all crawlers except AdsBot crawlers by using an asterisk ( * ) in your robots.txt file.

What does blocking Google-Extended affect?

Blocking Google-Extended in your robots.txt file prevents the crawlers that identify themselves with this token from requesting specified pages or files from your site. This directly impacts Google's ability to find and parse that content for display in Google Search and other Google properties, as Google uses various crawlers for its products. Google can index the content of most types of pages and files, but if a crawler is disallowed, it cannot request the content to process it.

It is important to note that Google's crawlers and fetchers typically only crawl the first 15MB of a file. Any content beyond this limit is ignored. Therefore, blocking access via robots.txt means that the content on the disallowed pages will not be processed by the Google-Extended crawlers, potentially limiting its visibility or utility within the Google products that rely on these specific crawlers.

What does blocking Google-Extended not affect?

Blocking Google-Extended through your robots.txt file does not serve as a form of access authorization or a substitute for valid content security measures. The IETF's Robots Exclusion Protocol clarifies that these rules are not a form of access authorization. It instructs compliant crawlers not to access certain parts of a site, but it does not technically prevent them from doing so, nor does it prevent other non-compliant programs or users from accessing the content.

Therefore, if you have sensitive content that you wish to keep private, relying solely on robots.txt to block Google-Extended or any other crawler is insufficient. For true content security, you must implement proper access controls and authentication mechanisms on your server.

Should you block Google-Extended?

The decision to block Google-Extended depends on your website's specific goals and how you want your content to be utilized by Google's products. If you want your content to be discoverable and used by Google's various properties, then allowing Google-Extended access is generally advisable. Blocking it would prevent the specific crawlers that use this token from accessing and processing your content for those purposes.

However, there might be situations where you consider blocking it. For instance, if your site is experiencing difficulties keeping up with Google's crawling requests, you have the option to reduce the crawl rate. Additionally, managing duplicate content through URL canonicalization can help avoid excessive crawling, which might be a concern for some sites. Ultimately, weigh the benefits of broader visibility across Google's ecosystem against any potential resource concerns or specific content control needs.

Common questions