AI crawlers and robots.txt: which bots to allow

What is the difference between AI search crawlers and training crawlers?

Each major provider now splits its crawling by purpose. One bot builds a search index so the product can link to you. Another collects content that may be used to train models. A third fetches a page only when a user asks for it. Robots.txt rules apply per bot, so you can allow one and block another.

This matters because many sites added broad AI blocks when the first training crawlers appeared. A rule written to keep content out of training can also keep the site out of AI search results if it names the wrong user agent or uses a wildcard.

What do GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot do?

These descriptions are taken from each company's own documentation, linked below.

• OAI-SearchBot (OpenAI): surfaces websites in ChatGPT's search features. Obeys robots.txt. OpenAI says a robots.txt change can take about 24 hours to take effect.
• GPTBot (OpenAI): crawls content that may be used to train generative AI models. Obeys robots.txt.
• ChatGPT-User (OpenAI): visits a page when a ChatGPT user or a Custom GPT asks for it. It is not an automatic crawler, and OpenAI says robots.txt rules may not apply to it.
• PerplexityBot (Perplexity): surfaces and links websites in Perplexity's search results, and is not used to crawl content for foundation models. Respects robots.txt.
• Perplexity-User (Perplexity): fetches a page when a user's question needs it. Perplexity says it generally ignores robots.txt because a user requested the fetch.
• Claude-SearchBot (Anthropic): indexes content to improve search results for Claude users. Respects robots.txt.
• ClaudeBot (Anthropic): collects web content that could contribute to model training. Respects robots.txt.
• Claude-User (Anthropic): accesses a site when a Claude user asks a question that needs it. Respects robots.txt.
• Google-Extended (Google): a control for whether content is used for AI training in other Google systems. AI Overviews and AI Mode rely on normal Search indexing, so they depend on Googlebot.

Which AI crawlers should I allow in robots.txt?

If your goal is to be cited, the safe default is to allow the search and user-requested bots and decide separately about training. A site with a single rule group, User-agent: * followed by Allow: /, already permits all of them. Problems come from extra groups added later. Check for a Disallow: / under GPTBot that was copied to OAI-SearchBot, a wildcard pattern that matches more than intended, or a firewall or bot-protection rule that blocks these user agents before robots.txt is even read.

If you want to stay out of training but remain in AI search, name the training bots only: disallow GPTBot, ClaudeBot and Google-Extended, and leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot allowed. That is a policy choice about your content, and the documentation supports making it per bot.

How do I check whether AI crawlers can reach my site?

Open your robots.txt and read every User-agent group, not only the first. Then request a page with each crawler's user agent string and confirm you get a 200 response with the real content, since a CDN or firewall can return a challenge page that robots.txt says nothing about. Finally, confirm the content is in the HTML. Text that appears only after client-side JavaScript runs may not be seen by a crawler that does not render.

Allowing a crawler is a condition for being cited, not a promise. dotmap's audit fetches your pages the way a crawler would and reports the ones that fail to load, are marked noindex or have thin text, which are the problems that keep a page out before content quality is even considered.