AI Bots: GPTBot, ClaudeBot, CCBot — Which Should You Allow
GPTBot is the crawler OpenAI uses for model training. Blocking it in robots.txt does not by itself remove your site from ChatGPT search; search uses a separate bot (OAI-SearchBot). The short answer: if you want visibility, allow the search bots; if you don’t want training use, block only the training bots. This page walks through what each bot does, gives copy-ready robots.txt examples, and lists the checks to run.
As of October 2026, the facts below come from each company’s own bot documentation (linked under the table). Bot names and behavior change, so check the linked page before you decide.
Quick Decision: Which Bots to Allow or Block
| What you want | Keep open | Can block |
|---|---|---|
| Appear in ChatGPT, Claude and Perplexity answers, but not be training data | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| Fine with both visibility and training use | All of them | None |
| Stay away from AI products entirely | Classic search bots such as Googlebot | All the AI bots above |
This table is not a visibility guarantee: allowing a bot means it can read your page, not that you’ll be chosen in answers.
What Each Bot Actually Does
An AI company usually runs more than one bot, and each does a different job: collecting training data, building a search index, or opening a single page when a user asks. Base your decision on those three jobs.
| Bot | Company | Job | How it treats robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train generative AI models | Blocking signals that your content should not be used for training |
| OAI-SearchBot | OpenAI | Crawls to show sites in ChatGPT search | If you block it, you are not shown in ChatGPT search answers (you can still appear as a navigational link) |
| ChatGPT-User | OpenAI | Opens a single page when a user asks; does not crawl automatically | Per OpenAI, because the action is user-initiated, robots.txt rules may not apply |
| ClaudeBot | Anthropic | Collects content that may contribute to model training | Blocking signals that your site’s future content should be excluded from training |
| Claude-SearchBot | Anthropic | Crawls to improve search result quality | If blocked, your content isn’t indexed for search |
| Claude-User | Anthropic | Accesses a site when a Claude user asks a question that needs it | If blocked, Claude can’t retrieve your content for user questions |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for model training | Managed through robots.txt |
| Perplexity-User | Perplexity | Visits a page when a user asks a question | Per Perplexity, it generally disregards robots.txt because the user requested the fetch |
| Google-Extended | Controls whether your content is used for Gemini training and for grounding in Gemini | Not a separate crawler, only a robots.txt token; doesn’t affect inclusion in Google Search | |
| Applebot-Extended | Apple | Controls whether data Applebot collected is used to train Apple’s generative models | Doesn’t crawl pages; blocking it doesn’t remove you from search |
| CCBot | Common Crawl (non-profit) | Crawls for an open, publicly available web archive | Honors robots.txt |
| Meta-ExternalAgent | Meta | Crawls to train AI models or index content directly | Can be blocked in robots.txt |
Sources (read October 2026): OpenAI bots · Anthropic bots · Perplexity bots · Google crawlers · Apple bots · CCBot · Meta crawlers.
Two things get mixed up often. Google-Extended is not a bot: per Google’s documentation it has no HTTP user-agent string of its own, so you won’t see it in server logs. It also doesn’t affect inclusion in Google Search, so blocking it won’t remove your site from Google results. The GEO Guide covers where these bots fit in the bigger picture.
Should I Allow GPTBot
If being training data doesn’t bother you, yes: allow it, and you don’t need to write anything. If it does, block GPTBot, but keep OAI-SearchBot open if you want to appear in ChatGPT search. OpenAI treats the two settings as independent; blocking GPTBot doesn’t change OAI-SearchBot’s search listing.
Blocking has a cost, too. For a site that doesn’t sell content (consulting, services, products) I see no concrete gain from blocking training, but there is a concrete cost to not being visible. I leave everything open on my own site: oguzerol.com/robots.txt contains only User-agent: * and Allow: /. Once you understand how GEO differs from traditional SEO, this choice tends to clarify itself. If your content is itself the product (news, course material), the call can differ, and blocking training bots is reasonable.
Copy-Ready robots.txt Examples
Each bot gets its own User-agent line. Add these to your own file.
Block training only, allow search bots:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Close only one section (for example a members area):
User-agent: GPTBot
Disallow: /members/
Allow: /One thing to watch: per Google’s documentation, a crawler follows only a single rule group, the most specific one matching its name. If you write a separate group for GPTBot, you may need to repeat the lines you wanted from User-agent: * inside it. That rule describes Google’s crawlers; for other companies’ bots, check their own documentation.
How to Check It in robots.txt
Finding out which bots your site allows takes two minutes, no coding knowledge required.
- Open
yoursite.com/robots.txtin your browser (example:oguzerol.com/robots.txt). - Look for
User-agent:lines. Each bot is listed in its own block by name. - If that block has
Disallow: /underneath it, the bot is banned from the entire site. IfDisallow:is empty, or there’s no block for that bot at all, the bot is free to crawl. - If a bot’s name doesn’t appear in the file at all, the general
User-agent: *rule at the top applies to it too. - If you’re on WordPress, robots.txt is usually generated virtually by Rank Math or Yoast. Don’t look for a physical file on the server; check the plugin’s robots.txt editor screen.
- If your site is behind Cloudflare, read the next section too; lines you never wrote may have been added.
- After a change, check the robots.txt report in Google Search Console. Google refreshes its cached copy about every 24 hours; to speed it up, the same report has a recrawl request (Google documentation). OpenAI and Perplexity also say a change can take about 24 hours to take effect.
Cloudflare’s AI Bot Blocking Setting
If your site is behind Cloudflare, your robots.txt may differ from what you wrote. When Cloudflare’s “managed robots.txt” feature is on, it prepends its own rules to your file and serves both as one; if you have no file, it serves only its own Disallow lines. Per its documentation, that block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent (Cloudflare documentation, read October 2026).
So looking at your own file isn’t enough: open your site’s live /robots.txt. Enforced blocking is handled by a separate tool (AI Crawl Control) (documentation). If you want search bots open, confirm separately that these settings don’t also shut out OAI-SearchBot and PerplexityBot; you can only see that in your own account settings, not in the documentation.
Is It the Real Bot: Verification
A user-agent line can be faked; anyone can put “GPTBot” in a request to piggyback on your rule. Before any critical decision, check where the request comes from.
- OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User (
openai.com/gptbot.json,openai.com/searchbot.json,openai.com/chatgpt-user.json). - Anthropic publishes its ranges at
claude.com/crawling/bots.json. The same page warns that blocking by IP may not guarantee an opt-out, because it stops the bots from reading your robots.txt. - Perplexity publishes separate IP lists for PerplexityBot and Perplexity-User.
- For CCBot there is reverse DNS (
crawl.commoncrawl.org) and a JSON range list.
When writing firewall rules, don’t pin to the version number in the bot string; OpenAI’s documentation gives GPTBot/1.4 as an example, but versions can change.
How One Line Can Affect a Whole Site
robots.txt is a small file but its impact can be hard to reverse. Google’s documentation says not to block resources a crawler needs to understand a page (Google documentation). Well-meaning Disallow rules on the folder holding the scripts or styles that build the page can leave a search engine seeing it incomplete.
I’ve seen something similar in my own operation: on one of the sites I manage, a plugin update reset the robots.txt output and wrote a broad Disallow rule over it; catching it could have taken weeks, and it was luckily caught during a routine check. Note: don’t confuse llms.txt with robots.txt. llms.txt only provides information and doesn’t block any bot; the access rule lives in robots.txt.
The reverse also holds: robots.txt is not a way to remove a page from results. Per Google, a blocked page’s URL can still appear without a description; to keep a page out you need noindex or password protection. The takeaway is simple: every time you touch robots.txt, check the live file right after the change, and know which folders a page actually needs before writing a broad Disallow.
Allowing Bots Isn’t Enough
Getting robots.txt right lets a bot crawl your site; being crawled doesn’t mean being visible. Once you grant access, the real question is whether your content is clear, quotable, and structured enough for the AI to pick you. I covered this in detail in How Your Website Shows Up in ChatGPT.
If you don’t want to deal with it yourself, my AI Search Visibility service audits and fixes the whole layer, from robots.txt to schema markup.
You Can Ask Me to Do This
You can ask me to do this: audit your robots.txt, determine which AI bots should be open or closed, and check whether an existing rule is accidentally blocking your site. It typically takes 2 hours to 2 weeks, fully remote, billed hourly. See the AI Search Visibility page for details, or write to me directly.
Frequently Asked Questions
Should I allow GPTBot?
If being training data doesn’t bother you, yes; you don’t need to write anything. If you don’t want training use, block GPTBot, but keep OAI-SearchBot open if you want to appear in ChatGPT search; OpenAI treats the two settings as independent.
If I block GPTBot, does that mean ChatGPT never shows my site?
No, two different bots are being conflated. GPTBot crawls for training content; ChatGPT’s search feature uses a separate bot, OAI-SearchBot. Per OpenAI’s documentation, sites opted out of OAI-SearchBot aren’t shown in ChatGPT search answers, while blocking GPTBot doesn’t affect that.
How long after changing robots.txt do AI bots notice?
OpenAI and Perplexity both say about 24 hours. Google also refreshes its cache about every 24 hours, and you can request a recrawl in Search Console. For other bots there’s no way to speed it up; it depends on that company’s crawl schedule.
Do ChatGPT-User and Perplexity-User follow robots.txt?
Both companies say these bots act on a user’s request, so robots.txt rules may not apply or are generally disregarded. You can’t stop them with robots.txt; if you need to restrict them, do it at the server or CDN level.
CCBot isn’t a search engine, so why does it matter?
CCBot doesn’t generate answers; it crawls for Common Crawl’s publicly available web archive, which others can use. Blocking it keeps you out of future crawls, but doesn’t remove content already archived.
Are bot permissions in robots.txt the same thing as llms.txt?
No. robots.txt is a formal access rule: compliant bots follow it, and some may not. llms.txt is just an informational/suggestion file and doesn’t block any bot. They do different jobs.
If I leave all AI bots open, does it load my server?
It can on very high-traffic sites. For a small-to-mid-sized WordPress site it is practically negligible; I haven’t seen any performance impact on oguzerol.com. If load becomes a problem, Anthropic supports the Crawl-delay extension.