AI Crawlers and Chatbots: What They Are and How to Control Them

A new category of web crawler has arrived. Alongside Googlebot and Bingbot, your logs now show visitors like GPTBot, ClaudeBot, PerplexityBot and Google-Extended. These belong to AI companies, and they're crawling the web to train large language models and to answer questions in AI chatbots. Understanding them helps you decide who gets access to your content.

What is a chatbot, really?

An AI chatbot like ChatGPT or Claude is a program that answers questions in natural language, powered by a large language model (LLM). To stay current and cite sources, many of them crawl the live web — either to train the model or to fetch pages in real time when a user asks something. That crawling is done by named bots with their own user-agents.

The main AI crawlers to know

• GPTBot — OpenAI's crawler for training data.
• OAI-SearchBot / ChatGPT-User — OpenAI fetching pages to answer live queries.
• ClaudeBot — Anthropic's crawler.
• PerplexityBot — Perplexity's search/answer crawler.
• Google-Extended — controls whether Google uses your content for its AI products (separate from Googlebot search).

Should you allow them?

There's no universally right answer — it's a business decision:

• Allow them if you want your brand and content surfaced in AI answers (many publishers and SaaS companies do — it's referral traffic and visibility).
• Block them if you don't want your content used for model training, or you sell access to that content.

Control access with robots.txt

Reputable AI crawlers honour robots.txt. To block training crawlers while keeping search visibility:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Still allow normal search engines
User-agent: Googlebot
Allow: /

To allow them instead, simply don't add a Disallow — the default is to permit crawling.

Enforce it at the server (for bots that ignore robots.txt)

robots.txt is a request, not a wall. To hard-block a crawler regardless, drop it at the web server:

if ($http_user_agent ~* "(GPTBot|ClaudeBot|PerplexityBot)") {
    return 403;
}

A note for the other side: being crawlable

If you want AI chatbots and search engines to read your site — for example, a docs site or blog — make sure your content is in the initial HTML (server-rendered or statically generated), not loaded only by JavaScript. Many AI crawlers don't execute JS, so client-only content is invisible to them.