Source: https://oguzerol.com/en/ai-bots/ GEO / AEO # AI Bots: GPTBot, ClaudeBot, CCBot — Which Should You Allow [Oğuz Erol](https://oguzerol.com/en/about-me/) | 11 min read | Published: 3 August 2026 | Updated: 13 October 2026 ## Quick summary How to write a GPTBot robots.txt rule: separate search bots from training bots, stay visible in ChatGPT while opting out of training. Copy-ready examples. GPTBot is the crawler OpenAI uses for model training. Blocking it in robots.txt does not by itself remove your site from ChatGPT search; search uses a separate bot (OAI-SearchBot). The short answer: **if you want visibility, allow the search bots; if you don’t want training use, block only the training bots.** This page walks through what each bot does, gives copy-ready robots.txt examples, and lists the checks to run. As of October 2026, the facts below come from each company’s own bot documentation (linked under the table). Bot names and behavior change, so check the linked page before you decide. ## Quick Decision: Which Bots to Allow or Block | What you want | Keep open | Can block | | --- | --- | --- | | Appear in ChatGPT, Claude and Perplexity answers, but not be training data | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | | Fine with both visibility and training use | All of them | None | | Stay away from AI products entirely | Classic search bots such as Googlebot | All the AI bots above | This table is not a visibility guarantee: allowing a bot means it can read your page, not that you’ll be chosen in answers. ## What Each Bot Actually Does An AI company usually runs more than one bot, and each does a different job: collecting training data, building a search index, or opening a single page when a user asks. Base your decision on those three jobs. | Bot | Company | Job | How it treats robots.txt | | --- | --- | --- | --- | | GPTBot | OpenAI | Crawls content that may be used to train generative AI models | Blocking signals that your content should not be used for training | | OAI-SearchBot | OpenAI | Crawls to show sites in ChatGPT search | If you block it, you are not shown in ChatGPT search answers (you can still appear as a navigational link) | | ChatGPT-User | OpenAI | Opens a single page when a user asks; does not crawl automatically | Per OpenAI, because the action is user-initiated, robots.txt rules may not apply | | ClaudeBot | Anthropic | Collects content that may contribute to model training | Blocking signals that your site’s future content should be excluded from training | | Claude-SearchBot | Anthropic | Crawls to improve search result quality | If blocked, your content isn’t indexed for search | | Claude-User | Anthropic | Accesses a site when a Claude user asks a question that needs it | If blocked, Claude can’t retrieve your content for user questions | | PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results; not used for model training | Managed through robots.txt | | Perplexity-User | Perplexity | Visits a page when a user asks a question | Per Perplexity, it generally disregards robots.txt because the user requested the fetch | | Google-Extended | Google | Controls whether your content is used for Gemini training and for grounding in Gemini | Not a separate crawler, only a robots.txt token; doesn’t affect inclusion in Google Search | | Applebot-Extended | Apple | Controls whether data Applebot collected is used to train Apple’s generative models | Doesn’t crawl pages; blocking it doesn’t remove you from search | | CCBot | Common Crawl (non-profit) | Crawls for an open, publicly available web archive | Honors robots.txt | | Meta-ExternalAgent | Meta | Crawls to train AI models or index content directly | Can be blocked in robots.txt | Sources (read October 2026): [OpenAI bots](https://developers.openai.com/api/docs/bots) · [Anthropic bots](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · [Perplexity bots](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) · [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) · [Apple bots](https://support.apple.com/en-us/119829) · [CCBot](https://commoncrawl.org/ccbot) · [Meta crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/). Two things get mixed up often. Google-Extended is not a bot: per Google’s documentation it has no HTTP user-agent string of its own, so you won’t see it in server logs. It also doesn’t affect inclusion in Google Search, so blocking it won’t remove your site from Google results. The [GEO Guide](https://oguzerol.com/en/geo-guide/) covers where these bots fit in the bigger picture. ## Should I Allow GPTBot If being training data doesn’t bother you, yes: allow it, and you don’t need to write anything. If it does, block GPTBot, but keep OAI-SearchBot open if you want to appear in ChatGPT search. OpenAI treats the two settings as independent; blocking GPTBot doesn’t change OAI-SearchBot’s search listing. Blocking has a cost, too. For a site that doesn’t sell content (consulting, services, products) I see no concrete gain from blocking training, but there is a concrete cost to not being visible. I leave everything open on my own site: `oguzerol.com/robots.txt` contains only `User-agent: *` and `Allow: /`. Once you understand [how GEO differs from traditional SEO](https://oguzerol.com/en/what-is-geo/), this choice tends to clarify itself. If your content is itself the product (news, course material), the call can differ, and blocking training bots is reasonable. ## Copy-Ready robots.txt Examples Each bot gets its own `User-agent` line. Add these to your own file. **Block training only, allow search bots:** ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / ``` **Close only one section** (for example a members area): ``` User-agent: GPTBot Disallow: /members/ Allow: / ``` One thing to watch: per Google’s documentation, a crawler follows only a single rule group, the most specific one matching its name. If you write a separate group for `GPTBot`, you may need to repeat the lines you wanted from `User-agent: *` inside it. That rule describes Google’s crawlers; for other companies’ bots, check their own documentation. ## How to Check It in robots.txt Finding out which bots your site allows takes two minutes, no coding knowledge required. 1. Open `yoursite.com/robots.txt` in your browser (example: `oguzerol.com/robots.txt`). 2. Look for `User-agent:` lines. Each bot is listed in its own block by name. 3. If that block has `Disallow: /` underneath it, the bot is banned from the entire site. If `Disallow:` is empty, or there’s no block for that bot at all, the bot is free to crawl. 4. If a bot’s name doesn’t appear in the file at all, the general `User-agent: *` rule at the top applies to it too. 5. If you’re on WordPress, robots.txt is usually generated virtually by Rank Math or Yoast. Don’t look for a physical file on the server; check the plugin’s robots.txt editor screen. 6. If your site is behind Cloudflare, read the next section too; lines you never wrote may have been added. 7. After a change, check the robots.txt report in Google Search Console. Google refreshes its cached copy about every 24 hours; to speed it up, the same report has a recrawl request ([Google documentation](https://developers.google.com/search/docs/crawling-indexing/robots/submit-updated-robots-txt)). OpenAI and Perplexity also say a change can take about 24 hours to take effect. ## Cloudflare’s AI Bot Blocking Setting If your site is behind Cloudflare, your robots.txt may differ from what you wrote. When Cloudflare’s “managed robots.txt” feature is on, it prepends its own rules to your file and serves both as one; if you have no file, it serves only its own Disallow lines. Per its documentation, that block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent ([Cloudflare documentation](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), read October 2026). So looking at your own file isn’t enough: open your site’s live `/robots.txt`. Enforced blocking is handled by a separate tool (AI Crawl Control) ([documentation](https://developers.cloudflare.com/ai-crawl-control/)). If you want search bots open, confirm separately that these settings don’t also shut out OAI-SearchBot and PerplexityBot; you can only see that in your own account settings, not in the documentation. ## Is It the Real Bot: Verification A user-agent line can be faked; anyone can put “GPTBot” in a request to piggyback on your rule. Before any critical decision, check where the request comes from. - OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User (`openai.com/gptbot.json`, `openai.com/searchbot.json`, `openai.com/chatgpt-user.json`). - Anthropic publishes its ranges at `claude.com/crawling/bots.json`. The same page warns that blocking by IP may not guarantee an opt-out, because it stops the bots from reading your robots.txt. - Perplexity publishes separate IP lists for PerplexityBot and Perplexity-User. - For CCBot there is reverse DNS (`crawl.commoncrawl.org`) and a JSON range list. When writing firewall rules, don’t pin to the version number in the bot string; OpenAI’s documentation gives GPTBot/1.4 as an example, but versions can change. ## How One Line Can Affect a Whole Site robots.txt is a small file but its impact can be hard to reverse. Google’s documentation says not to block resources a crawler needs to understand a page ([Google documentation](https://developers.google.com/search/docs/crawling-indexing/robots/intro)). Well-meaning `Disallow` rules on the folder holding the scripts or styles that build the page can leave a search engine seeing it incomplete. I’ve seen something similar in my own operation: on one of the sites I manage, a plugin update reset the robots.txt output and wrote a broad Disallow rule over it; catching it could have taken weeks, and it was luckily caught during a routine check. Note: don’t confuse [llms.txt](https://oguzerol.com/en/what-is-llms-txt-and-how-do-you-build-one/) with robots.txt. llms.txt only provides information and doesn’t block any bot; the access rule lives in robots.txt. The reverse also holds: robots.txt is not a way to remove a page from results. Per Google, a blocked page’s URL can still appear without a description; to keep a page out you need `noindex` or password protection. The takeaway is simple: every time you touch robots.txt, check the live file right after the change, and know which folders a page actually needs before writing a broad `Disallow`. ## Allowing Bots Isn’t Enough Getting robots.txt right lets a bot crawl your site; being crawled doesn’t mean being visible. Once you grant access, the real question is whether your content is clear, quotable, and structured enough for the AI to pick you. I covered this in detail in [How Your Website Shows Up in ChatGPT](https://oguzerol.com/en/how-your-website-shows-up-in-chatgpt/). If you don’t want to deal with it yourself, my [AI Search Visibility](https://oguzerol.com/en/ai-search-visibility-geo-aeo/) service audits and fixes the whole layer, from robots.txt to schema markup. ## You Can Ask Me to Do This **You can ask me to do this:** audit your robots.txt, determine which AI bots should be open or closed, and check whether an existing rule is accidentally blocking your site. It typically takes 2 hours to 2 weeks, fully remote, billed hourly. See the [AI Search Visibility](https://oguzerol.com/en/ai-search-visibility-geo-aeo/) page for details, or write to me directly. ## Frequently Asked Questions ### Should I allow GPTBot? If being training data doesn’t bother you, yes; you don’t need to write anything. If you don’t want training use, block GPTBot, but keep OAI-SearchBot open if you want to appear in ChatGPT search; OpenAI treats the two settings as independent. ### If I block GPTBot, does that mean ChatGPT never shows my site? No, two different bots are being conflated. GPTBot crawls for training content; ChatGPT’s search feature uses a separate bot, OAI-SearchBot. Per OpenAI’s documentation, sites opted out of OAI-SearchBot aren’t shown in ChatGPT search answers, while blocking GPTBot doesn’t affect that. ### How long after changing robots.txt do AI bots notice? OpenAI and Perplexity both say about 24 hours. Google also refreshes its cache about every 24 hours, and you can request a recrawl in Search Console. For other bots there’s no way to speed it up; it depends on that company’s crawl schedule. ### Do ChatGPT-User and Perplexity-User follow robots.txt? Both companies say these bots act on a user’s request, so robots.txt rules may not apply or are generally disregarded. You can’t stop them with robots.txt; if you need to restrict them, do it at the server or CDN level. ### CCBot isn’t a search engine, so why does it matter? CCBot doesn’t generate answers; it crawls for Common Crawl’s publicly available web archive, which others can use. Blocking it keeps you out of future crawls, but doesn’t remove content already archived. ### Are bot permissions in robots.txt the same thing as llms.txt? No. robots.txt is a formal access rule: compliant bots follow it, and some may not. llms.txt is just an informational/suggestion file and doesn’t block any bot. They do different jobs. ### If I leave all AI bots open, does it load my server? It can on very high-traffic sites. For a small-to-mid-sized WordPress site it is practically negligible; I haven’t seen any performance impact on oguzerol.com. If load becomes a problem, Anthropic supports the Crawl-delay extension. [Back to top ↑](https://oguzerol.com/en/ai-bots/#icerik) ## You may also find these useful [What Is llms.txt, and How Do You Build One?What is llms.txt, and how does it differ from robots.txt? Automatic setup with Rank Math on WordPress, what to put in it — an honest walkthrough.](https://oguzerol.com/en/what-is-llms-txt-and-how-do-you-build-one/)[GEO Guide: Visibility in AI SearchWhat actually helps with GEO and visibility in AI search, Google's official position, and a reading order for 20 articles: understand, apply, measure.](https://oguzerol.com/en/geo-guide/)[Preparing Your Site for AI SearchIf AI can't read your site, it can't mention you. I set up llms.txt, crawler permissions, entity schema, and citable content structure. Get in touch.](https://oguzerol.com/en/preparing-your-site-for-ai-search/) [Browse all articles →](https://oguzerol.com/en/articles/)