How to use sitemap index files for large websites

What is a sitemap index file?

A sitemap is a file that provides information about the pages, videos, and other files on your website, as well as the relationships between them. Search engines read these files to crawl your site more efficiently. A sitemap index file serves as a master list, containing the locations of multiple individual sitemap files. Its XML format is similar to that of a regular sitemap file, and it allows you to submit many sitemaps at once. Google notes that while sitemaps help search engines discover URLs, they do not guarantee that all items listed will be crawled and indexed.

Why do large websites need sitemap index files?

Large websites often exceed the capacity of a single sitemap file. The Sitemap Protocol specifies that each sitemap file must contain no more than 50,000 URLs and be no larger than 50MB. Bing also supports this limit of 50,000 URLs per individual sitemap file. If your site has more than 50,000 URLs, you must create multiple sitemap files. Google explains that on large sites, it is more challenging to ensure every page is linked internally, which can lead to Googlebot not discovering some new pages. Sitemaps, including those organized by an index file, can improve the crawling of larger or more complex sites, and help search engines find specialized content like video and image files or news articles.

What are the size limits for sitemap index files?

Just like individual sitemap files, sitemap index files also have size limitations. A sitemap index file may not list more than 50,000 sitemaps and must be no larger than 50MB. These files can be compressed to save space. Google specifies that a sitemap index file may have up to 50,000 'loc' tags, which refer to individual sitemaps. Bing notes that a single sitemap index file can reference up to 2.5 billion URLs. For extremely large domains, Bing states that multiple index files can support up to 2.5 trillion URLs. Google allows site owners to submit up to 500 sitemap index files for each site in their Search Console account.

How do sitemap index files help search engines crawl more efficiently?

Sitemap index files can include a 'lastmod' timestamp for each sitemap listed. This feature, documented by Sitemaps.org, allows search engine crawlers to retrieve only a subset of the sitemaps in the index. For example, a crawler might only fetch sitemaps that were modified since a specific date. This incremental fetching mechanism is crucial for the rapid discovery of new URLs on very large sites. Bing highlights that the 'lastmod' field is a key signal, helping it prioritize URLs for recrawling and reindexing, or to skip them if the content has not changed since the last crawl. Bing advises using standard ISO 8601 date and time formatting for 'lastmod' values and to avoid setting 'lastmod' to the sitemap generation time unless the content on that URL was actually updated.

Where should sitemap index files and their sitemaps be located?

The Sitemap Protocol specifies that a sitemap index file can only list sitemaps that are found on the same site as the sitemap index file itself. Google further clarifies this by stating that the referenced sitemaps must be hosted on the same site. Additionally, Google recommends that sitemaps referenced in the sitemap index file should be in the same directory as the sitemap index file, or lower in the site hierarchy. This organizational structure helps search engines correctly associate the sitemaps with the main site.

How do you submit a sitemap index file to search engines?

Once you have created your sitemap index file, you can submit its location to search engines. The Sitemap Protocol states that if you have a sitemap index file, you can include the location of just that file. Bing notes that once your sitemap is referenced in your robots.txt file or submitted through Bing Webmaster Tools, Bing will attempt to fetch it immediately upon submission. If you encounter any issues with your sitemap, Google suggests using Google Search Console to investigate errors.

Common questions