What are Googlebot-Image and Googlebot-Video, and how do they work?

What are Googlebot-Image and Googlebot-Video?

Google documents that a crawler, which is also called a robot or spider, is a generic term for any program used to automatically discover and scan websites. Google uses common crawlers to find information for building search indexes, running product-specific crawls, and performing technical analysis. Google's crawlers and fetchers support both HTTP/1.1 and HTTP/2 network protocols.

Crawling preferences addressed to the Googlebot-Image user agent control how images appear across Google services. This user agent affects Google Images, Discover, Video mode, and any feature in Google Search where images, logos, and favicons are presented. Crawling preferences set for the Googlebot-Video user agent affect video-related Google Search features and other products that depend on video content.

How do Google media crawlers fetch files?

Google aims to crawl as many pages from a site as possible during each visit without overwhelming the web server. By default, Google's crawlers and fetchers only crawl the first 15MB of a file. Any content beyond this 15MB limit is ignored.

When crawling files for Google Search, Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF file. Other specialized Google crawlers, such as Googlebot-Image and Googlebot-Video, may operate under different size limits. All of these tools make their HTTP requests over HTTP/1.1 or HTTP/2.

How do you control access using robots.txt?

A robots.txt file tells search engine crawlers which URLs they are allowed to access on a website. Google explains that a robots.txt file is used primarily to manage crawler traffic to a site and keep files off Google, depending on the file type. Google's common crawlers always obey rules placed in your robots.txt file during automatic crawls.

Some crawlers use more than one user agent token. A site owner only needs to match one crawler token for a rule to apply. Blocking the main Googlebot user agent affects Google Search, Discover, Google Images, Google Video, and Google News.

While Googlebot and other respectable web crawlers follow the instructions in a robots.txt file, unverified crawlers might not. If content is blocked by a robots.txt file, Google will not crawl or index the content itself, but Google might still find and index a disallowed URL if it is linked from other places on the web.

Why must video and thumbnail files stay accessible?

Google needs to successfully fetch the actual bytes of a video file to enable features like video previews and key moments. To make your videos eligible for these search features, you must allow Google to fetch your video files.

Site owners should not block the URL of the actual video bytes with a noindex rule or a disallow rule in a robots.txt file. Additionally, thumbnail files must remain accessible to Googlebot and Googlebot-Image. Blocking a thumbnail file using a robots.txt file or putting it behind a login requirement prevents Google from displaying media previews.

How do you troubleshoot media crawler issues?

To ensure Googlebot-Image and Googlebot-Video can access and scan your assets correctly, follow these diagnostic steps to verify access control and request authenticity.

1. 01Inspect your robots.txt file to ensure image URLs, video byte URLs, and thumbnail files are not blocked for Googlebot, Googlebot-Image, or Googlebot-Video.
2. 02Verify that no video URLs or thumbnail assets are restricted by a noindex meta tag or hidden behind a login requirement.
3. 03Confirm that requests sent to your server under the Googlebot user-agent strings are legitimate by matching the source IP address against official Googlebot IP ranges or by executing a reverse DNS lookup.

Common questions