What is a user-agent, and how do search engines use it?
guide // , 4 min read, by Darthwares
A user agent is a computer program that represents a person and identifies itself to a web server. Search engines use these self-identifying strings to manage how their automated programs, such as crawlers, interact with websites. This identification helps webmasters control access to their content and understand who is visiting their pages.
On this page // 5 sections
The short version
- A user agent is a program that accesses the web on behalf of a person, such as a browser or a bot.
- User-agent strings are HTTP headers that identify the program, its version, and its operating system to a web server.
- Search engines use specific user-agent strings to identify their crawlers and fetchers.
- The Robots Exclusion Protocol uses user-agent strings in robots.txt files to define access rules for different automated clients.
- Some programs may send fake user-agent strings to misrepresent themselves.
What is a user-agent?
A user agent is a computer program that acts on behalf of a person when accessing the web. MDN Web Docs explains that this program could be a web browser, a bot that scrapes webpages, a download manager, or any other application that connects to the internet.
When a user agent makes a request to a server, it includes a self-identifying User-Agent HTTP header, also known as a user agent (UA) string. This string typically provides details about the client, such as the browser type, its version number, and the operating system it is running on. For example, a common UA string might look like "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0".
Developers can access the user agent string on the client side using JavaScript through the navigator.userAgent property. However, it is important to note that some programs, including spam bots, download managers, and certain browsers, may send a fake UA string to present themselves as a different client.
How do search engines use user-agents?
Search engines use user-agent strings to identify their automated programs that interact with websites. Google, for instance, uses various crawlers and fetchers to perform actions for its products. A crawler, sometimes called a "robot" or "spider," is a general term for any program that automatically discovers and scans websites. Fetchers, on the other hand, typically make a single request on behalf of a user, similar to a program like wget. Googlebot is one of the common crawlers used for Google's products.
Google's crawlers identify themselves in three distinct ways: through the HTTP user-agent request header, by the source IP address of the request, and via the reverse DNS hostname of the source IP. Google documents that its crawlers and fetchers typically crawl only the first 15MB of a file, ignoring any content beyond this limit.
How do robots.txt files use user-agents?
The Robots Exclusion Protocol (REP) is a method for website owners to control how automated clients, known as crawlers, may access their content. The RFC Editor specifies that this protocol, originally defined in 1994, outlines rules that crawlers are requested to honor when accessing pages. Search engines, for example, use crawlers to traverse links recursively for indexing purposes.
The protocol's rules are made available in a file named "robots.txt" located at the root of a website. This file consists of groups of rules, each typically starting with a "User-agent" directive. The RFC Editor specifies that the product token in this directive should be a substring of the identification string that the crawler sends to the service.
Crawlers must use case-insensitive matching to find the group that corresponds to their product token and then obey the rules within that group. To determine if access to a specific page is allowed, a crawler must match the paths defined in "allow" and "disallow" rules against the page's URL. Google documents that its crawlers always respect these robots.txt rules for automatic crawls.
How can you verify a search engine crawler?
Google's crawlers identify themselves through three specific pieces of information: the HTTP user-agent request header, the source IP address of the request, and the reverse DNS hostname of the source IP. Google documents that these details can be used to verify the authenticity of its crawlers and fetchers, helping webmasters ensure that the programs accessing their site are legitimate Google services.
Common questions
What is a crawler?
A crawler is a generic term for any computer program that is used to automatically discover and scan websites. Search engines use crawlers to traverse links and gather information for indexing, as specified by the RFC Editor.
What is a fetcher?
Google Developers explain that fetchers act as a program like wget, typically making a single request on behalf of a user. These are distinct from crawlers, which are designed for automatic and recursive discovery of websites.
Can a user-agent string be faked?
Yes, MDN Web Docs states that spam bots, download managers, and some browsers often send a fake user agent string. This is done to announce themselves as a different client than they actually are.
How much content do Google's crawlers process?
Google Developers document that by default, Google's crawlers and fetchers only crawl the first 15MB of a file. Any content that extends beyond this limit is ignored by these programs.
How do Google's crawlers identify themselves?
Google's crawlers identify themselves in three ways: through the HTTP user-agent request header, by the source IP address of the request, and via the reverse DNS hostname of the source IP. These details can be used to verify their authenticity.
Primary sources and further reading
This article was researched and written by an AI model and published automatically. It was written from sentences that were checked to appear on the pages below, but a person has not reviewed the text. If something is wrong, tell us at chibivikramathithan@darthwares.com and we will correct it.