Block or Allow AI Crawlers? Why It's Not a Single Switch

In 2026, a growing share of the traffic to your website no longer comes from people, but from bots. Many react on reflex: get the AI crawlers out, block everything, be done with it.

The reflex is understandable. It just tends to backfire. "Blocking AI crawlers" sounds like a switch you flip. In reality it's several bots with different jobs. Block the wrong one and you vanish from AI answers where you actually want to appear. Block another one thinking you're shutting out AI search, and nothing happens there at all.

In this post we show you which bot does what, which two misconceptions are the most expensive, and which three questions get you to a clean decision on your own. No hype, with a look at the providers' official documentation.

What it's really about: three types of crawlers

A crawler (a program that automatically fetches and reads websites) isn't just a crawler. With AI services, it helps to sort them into three roles.

1. Training crawlers. They collect text so an AI model can learn from it. Examples: Google-Extended (for Gemini) and GPTBot (for OpenAI's models). Block them and your content isn't used for training. That doesn't touch your visibility in normal search.

2. Search crawlers. They decide whether your page shows up as a source in an AI answer. Examples: Googlebot (the basis for the AI Overviews) and OAI-SearchBot (for ChatGPT search). As a rule you want to let these bots in if you want to be visible in AI answers.

3. On-demand crawlers. They fetch a page exactly when a user requests it. Example: ChatGPT-User, which retrieves a URL someone pastes into the chat. This isn't mass crawling, it's a single fetch at the user's request.

The crucial point: these roles aren't connected. Blocking a training crawler says nothing about your visibility in AI search. This is exactly where the expensive mistakes happen.

How to actually control crawlers: robots.txt

Control runs through robots.txt. It's a small text file in the root directory of your domain (reachable at your-domain.com/robots.txt). In it you tell each bot what it's allowed to do.

Here's an example that locks out training while keeping search visibility:

``` # Block training crawlers User-agent: Google-Extended Disallow: /

User-agent: GPTBot Disallow: /

User-agent: CCBot Disallow: /

# Explicitly allow search crawlers User-agent: OAI-SearchBot Allow: / ```

Two honest notes. First: whatever you don't block stays allowed. So you don't need to explicitly permit Googlebot; without a disallow it's allowed anyway. Second, and more important: robots.txt is a request, not a fence. Reputable providers like Google and OpenAI honor it. But it's not a technical access block. Anyone who really has to keep unwanted bots out also needs a firewall or a block at the server level.

Misconception 1: "Blocking Google-Extended hides me from AI search"

This is the most common error, and it costs you either visibility or nerves, depending.

Google-Extended isn't a bot of its own crawling around. It's purely a control token in robots.txt. With it you set whether Google may use your content to train and ground its Gemini models (grounding means pulling in current content for the answer). Google states clearly in its documentation: Google-Extended has no effect on whether a page is included in Google Search and is not a ranking signal.

And the AI Overviews (the AI answer at the top of Google's results)? Those run through normal search, per Google. The condition for appearing there as a source: your page is indexed and eligible for a snippet. That's controlled through Googlebot and the usual snippet settings, not through Google-Extended.

Plainly: if you block Google-Extended, you protect your text from Gemini training. It doesn't throw you out of the AI Overviews. Anyone who thinks this one line makes them vanish from AI search is wrong. And anyone avoiding it out of fear for their ranking can relax.

Misconception 2: "Blocking GPTBot stops my citations in ChatGPT"

The second misconception mirrors the first, just at OpenAI.

OpenAI separates its bots cleanly. GPTBot collects content for model training. OAI-SearchBot is the bot that makes pages visible as a source in ChatGPT search. And ChatGPT-User fetches a page when a user requests it directly.

Now the consequence. Block GPTBot and you only stop training. That doesn't touch your citations in ChatGPT search. Those hang on OAI-SearchBot. OpenAI puts it plainly: pages that exclude OAI-SearchBot are not shown in ChatGPT search answers.

From that follows a clear rule. Want to protect your material from training but still be cited in ChatGPT search? Then block GPTBot and allow OAI-SearchBot. Block both wholesale and you give away the visibility that, for many B2B companies, is the actual reason to think about AI search at all.

Perplexity, Common Crawl, and the rest

Two more names come up often. PerplexityBot fetches content for Perplexity's answers. Per their help pages, it obeys robots.txt. Block it and your chance of appearing in Perplexity answers drops.

CCBot belongs to Common Crawl, an open web dataset that many AI projects use as raw material. CCBot follows robots.txt too. Block it and your content doesn't land in the Common Crawl archive.

There are more bots, and the list changes. That's why memorizing them is the wrong path. The right path is a principle: assign each bot to one of the three roles (training, search, on-demand retrieval). Then you know what a block does.

The three questions that really decide it

Feature lists and bot names mislead. In practice, three plain questions decide it. This is the grid we run projects through.

1. Do you want to keep your content from training AI models? If yes, block the training crawlers: Google-Extended, GPTBot, CCBot. That's a matter of stance, not an SEO lever.
2. Do you want to stay visible in AI answers? If yes, let the search crawlers in: Googlebot for the AI Overviews, OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity. Wanting both at once (no training, but visibility) is entirely legitimate and technically possible.
3. Is this really about server load? If your trigger is heavy bot traffic, robots.txt only half solves it. Load is better throttled through server rules or a firewall. That's a different job than the visibility question.

Answer these three questions honestly and the robots.txt almost writes itself.

What this means for the B2B mid-market

For many mid-sized companies the second question is the most important. Anyone in sales who depends on inquiries from search wants to show up in AI answers, not vanish from them. For them: keep the search crawlers open, and take your time on the training decision.

It's different when your own expert content is a real asset, say elaborate studies or proprietary methods. Then it can make sense to block training and still stay visible in search. Exactly this combination is possible, but it's rarely set up cleanly.

And one more point that often gets lost: robots.txt isn't a one-time thing. New bots appear, old ones change their names. A look once a quarter is usually enough. But with no upkeep at all the file goes stale, like any background technology.

What we take from this

An honest word from practice. We've handled robots.txt and crawling since 2007, across TYPO3, WordPress, and Neos. What we keep seeing: most "block AI" decisions are made on gut feeling, not from a clear question.

The pattern is almost always the same. Someone reads a headline about AI crawlers, copies a robots.txt from a forum, and blocks everything that sounds like AI wholesale. Weeks later they wonder why the company no longer shows up in ChatGPT search. The reason is right there in the copied file.

The calmer way is unspectacular: first clarify what you want to protect, then set the right bots. Not the other way around.

Conclusion

Block or allow AI crawlers: the answer isn't in one line, it's in your intent. Training crawlers like Google-Extended and GPTBot control whether your text feeds AI models. Search crawlers like Googlebot and OAI-SearchBot control whether you're visible in AI answers. Those are two separate questions.

Go through the three questions. They bring you closer to the right robots.txt than any ready-made template from the web. And if you're unsure whether your current setup protects you or makes you invisible, a quick look can settle it.

→ Have your website's robots.txt and AI crawler setup reviewed

• developers.google.com: Google's common crawlers. developers.google.com
• developers.google.com: AI Features and Your Website. developers.google.com
• developers.google.com: Robots.txt Introduction/Refresher. developers.google.com
• developers.openai.com: Overview of OpenAI Crawlers. developers.openai.com
• perplexity.ai Help Center: How Perplexity Follows robots.txt. perplexity.ai
• commoncrawl.org: CCBot and robots.txt. commoncrawl.org