AI Bots: GPTBot, ClaudeBot, CCBot — Which Should You Allow

GPTBot, ClaudeBot, CCBot, and similar bots are what AI systems like ChatGPT, Claude, and Perplexity use to crawl the web. Some collect training data, some pull pages to generate real-time answers. Whether you allow these bots in your robots.txt file determines whether your site ever shows up in AI search (ChatGPT, Perplexity, Google AI Overviews) at all — think of it as the AI equivalent of Googlebot in SEO, except with five different companies and five different rules.

What Each Bot Actually Does

Every AI company has its own bot, and each crawls for a different purpose — some train the model, some fetch pages for real-time answers. Don’t mix them up; each requires a different decision.

This list is the technical layer of GEO (Generative Engine Optimization); if you want the full picture of the concept, continue with the GEO Guide.

How to Check It in robots.txt

Finding out which bots your site allows takes two minutes, no coding knowledge required.

1. Open yoursite.com/robots.txt in your browser (example: oguzerol.com/robots.txt).
2. Look for User-agent: lines in the file — each bot is listed in its own block by name (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, OAI-SearchBot).
3. If that block has Disallow: / underneath it, the bot is banned from the entire site. If Disallow: is empty, or there’s no block for that bot at all, the bot is free to crawl.
4. If a bot’s name doesn’t appear in the file at all, the general User-agent: * rule at the top applies to it too.
5. If you’re on WordPress, robots.txt is usually generated virtually by Rank Math or Yoast — don’t look for a physical file on the server, check the plugin’s robots.txt editor screen instead.
6. After making a change, verify it with Google Search Console’s robots.txt testing tool — at minimum you’ll confirm it’s error-free for Googlebot.

Should You Keep It Open or Closed

There’s no single right answer, but the decision criteria are clear.

If AI visibility is your priority: keep all of them open — especially OAI-SearchBot, PerplexityBot, and Google-Extended. These currently generate real answers for real users and can show your site as a source; blocking them removes you from that traffic entirely.

If you’re uncomfortable with competitors “learning” from your content: you could consider blocking only the pure training bots (GPTBot, ClaudeBot, CCBot). But make this decision knowing it has no measurable commercial harm today — it’s a matter of principle. Your content is already public and anyone can read it; the difference is whether the model “memorizes” it.

If you’re looking for a middle ground: keeping search bots open and training bots closed sounds logical, but in practice the line is blurry — some bots, like ClaudeBot, use the same user-agent for both training and live access, so you can’t separate the two at the robots.txt level.

My own rule: for a site whose primary income isn’t content sales (consulting, services, products), there’s no concrete gain from blocking, but there’s a concrete cost to not being visible. Once you understand how GEO differs from traditional SEO, this choice tends to clarify itself. I leave all of them open by default.

One Line Can Wipe Out an Entire Site

robots.txt is a small file but its impact can be hard to reverse. A case discussed this week showed exactly this: a site added its build-output folder /_next/ as Disallow in robots.txt. Since the files in that folder are the JavaScript pieces that “build” the page in the browser, search engines started seeing the page as empty — the result was nearly the entire site dropping out of search. One well-intentioned, single-line rule (probably added to keep “unnecessary files” out of crawling) removed a whole site from the index.

I’ve seen something similar in my own operation: on one of the sites I manage, a plugin update reset the robots.txt output and wrote a broad Disallow rule over it; catching it could have taken weeks — it was luckily caught during a routine check. Note: don’t confuse llms.txt with robots.txt — llms.txt only provides information, it doesn’t block any bot; the binding access rule is always in robots.txt.

The takeaway is simple: every time you touch robots.txt, verify the change with the testing tool before pushing it live, and know which folders are actually needed for rendering before writing a broad Disallow.

Allowing Bots Isn’t Enough

Getting robots.txt right lets a bot crawl your site — being crawled doesn’t mean being visible. Once you grant access, the real question becomes: is that content clear, quotable, and structured enough for the AI to pick you as the answer? I covered this in detail in How Your Website Shows Up in ChatGPT.

If you don’t want to deal with it yourself, my AI Search Visibility service audits and fixes the whole layer, from robots.txt to schema markup.

You Can Ask Me to Do This

You can ask me to do this: audit your robots.txt, determine which AI bots should be open/closed, and check whether an existing rule is accidentally blocking your site — work that typically takes 2 hours to 2 weeks, fully remote, billed hourly. See the AI Search Visibility page for details, or write to me directly.

Frequently Asked Questions

How long after changing robots.txt do AI bots notice?

It varies. If you submit a “recrawl” request in Google Search Console, Googlebot and Google-Extended can notice within days. There’s no manual way to speed things up for bots like GPTBot, ClaudeBot, or PerplexityBot — they revisit on their own crawl schedules, ranging from a few days to a few weeks.

If I block GPTBot, does that mean ChatGPT never shows my site?

No, two different bots are being conflated. GPTBot collects training data; ChatGPT’s live search/answer feature uses a separate bot, OAI-SearchBot. Even if you block GPTBot, ChatGPT’s search mode can still show your site if OAI-SearchBot is open.

CCBot isn’t a search engine, so why does it matter?

CCBot itself doesn’t generate answers, but the Common Crawl archive it collects is part of the initial training set for most large language models. Blocking it today doesn’t change what a model currently “remembers,” but it can prevent future model versions from learning about you again.

Are bot permissions in robots.txt the same thing as llms.txt?

No. robots.txt is a formal access rule — bots comply with it (or some don’t, but that’s the rule). llms.txt is just an informational/suggestion file, it doesn’t block any bot. They do different jobs; doing one doesn’t make the other unnecessary.

If I leave all AI bots open, does it load my server?

For very high-traffic large sites, yes — CCBot and GPTBot in particular can create a noticeable crawl load. For a small-to-mid-sized WordPress site this is practically negligible — I haven’t seen any performance impact on oguzerol.com.