# robots.txt and AI bots: a guide to GPTBot, ClaudeBot, PerplexityBot and Google-Extended > What OpenAI, Anthropic, Perplexity and Google bots do, from official docs: a sample robots.txt that blocks training and stays visible in AI search. Published: 2026-10-07 robots.txt is a plain text file that sits in your site's root directory and tells crawlers which paths you ask them not to crawl; AI companies' bots read this file too. Because OpenAI, Anthropic and Perplexity use separate bot names for search, user requests and model training, robots.txt lets you turn off collection for training while staying visible in these engines' search. With Google the split works differently: Google-Extended does not affect Search, but it is not limited to training either. Every bot name and role description in this guide was re-read from the companies' own documentation on 7 October 2026. These documents are updated often; we recommend checking the links once more before you write a rule. ## robots.txt is a request, not a lock robots.txt is not access control. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) (September 2022), the protocol's standard document, states explicitly that these rules are not a form of access authorization, and asks crawlers to honour them. The companies named below recommend in their own documentation that you manage their search and training bots with robots.txt; the information that a bot complies with these rules is not a legal obligation but the company's own statement. If you need a hard block, the rule is written on the server, in the firewall or in the CDN. robots.txt, by contrast, is the simplest way to tell documented bots what you want, and it is enough for most decisions. Two rules of the protocol explain most of the mistakes made with AI bots: - **Which group applies?** A bot applies the group in robots.txt that names it. If there is no group specific to its name, it follows the wildcard group (`User-agent: *`). If there is no wildcard group either, no restriction applies to it. - **When does a change take effect?** RFC 9309 says crawlers should generally not use a cached copy of robots.txt for more than 24 hours. OpenAI also writes that a robots.txt change for OAI-SearchBot may take about 24 hours to be reflected in its systems. ## Three kinds of AI bot The requests an AI company sends to your site serve one of three roles. This distinction is the basis of the robots.txt decision: | Role | What it does | What happens if you block it | |---|---|---| | Search index | Reads pages in advance and adds them to the engine's search | The path to appearing as a source in that engine's search answers closes | | User request | Opens the page on the spot when a user asks a question or gives an address | The engine cannot read your page at the moment the user asks for it | | Model training | Collects content that may be used to train future models | You have asked for your content not to be taken into training data | A common mistake is to write a blanket `Disallow: /` in the wildcard group while trying to stop training bots, and so shut out the search bots too. If search bots are not in a group under their own names, they follow the wildcard rule. ## Bots company by company: a table from the official documentation The table below shows the bots listed in the four companies' documentation and their roles. The "Does robots.txt apply?" column is the company's own statement. | Bot name | Company | Role | Does robots.txt apply? | |---|---|---|---| | OAI-SearchBot | OpenAI | Search index (ChatGPT search features) | Yes | | ChatGPT-User | OpenAI | User request | According to OpenAI, it may not apply | | GPTBot | OpenAI | Model training | Yes | | Claude-SearchBot | Anthropic | Search index | Yes | | Claude-User | Anthropic | User request | Yes, according to Anthropic's statement | | ClaudeBot | Anthropic | Model training | Yes | | PerplexityBot | Perplexity | Search index | Yes, Perplexity recommends managing it with robots.txt | | Perplexity-User | Perplexity | User request | According to Perplexity, generally does not apply | | Googlebot | Google | Google Search index | Yes | | Google-Extended | Google | Not a separate bot but a control token: Gemini training and grounding in Gemini Apps | Yes, only as a name in robots.txt | ### OpenAI According to [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), OAI-SearchBot is used to show sites in ChatGPT's search features. Sites that block OAI-SearchBot are not shown in ChatGPT search answers; the documentation adds that such sites may still appear as navigational links. GPTBot is used for OpenAI's generative AI foundation models. Blocking GPTBot signals that the site's content should not be used to train these models. OpenAI also writes that if both bots are allowed, it may use the results of a single crawl for both purposes to avoid duplicate crawling. ChatGPT-User opens pages for certain user actions in ChatGPT and in custom GPTs. OpenAI states that because these actions are initiated by the user, robots.txt rules may not apply, and that ChatGPT-User is not used to determine whether content appears in search. For search preferences, it recommends managing OAI-SearchBot. ### Anthropic [Anthropic's help page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (updated 7 April 2026) defines three bots. ClaudeBot collects web content that may contribute to training; restricting ClaudeBot signals that the site's future content should be excluded from training datasets. Claude-SearchBot browses the web to improve the quality of users' search results; if it is blocked, content is not indexed for search. Claude-User, on the other hand, accesses a site when a user asks Claude a question; if it is blocked, Claude cannot retrieve your content when answering the user's question, and the page may, in Anthropic's words, be less visible in user-directed search. Anthropic writes that its bots comply with the standard directives in robots.txt and also support the non-standard `Crawl-delay` extension. Each bot is managed separately: blocking ClaudeBot does not block Claude-SearchBot or Claude-User. The page also warns that blocking IP addresses may not be a reliable way to state a preference, because it also stops the bot from reading robots.txt. ### Perplexity According to [Perplexity's documentation](https://docs.perplexity.ai/guides/bots), PerplexityBot is designed to surface and link sites in Perplexity's search results and is not used to crawl content for AI foundation models. Perplexity recommends allowing PerplexityBot in robots.txt for sites to appear in search, and writes that robots.txt changes may take up to 24 hours to be reflected in its systems. Perplexity-User may visit a page to give an accurate answer when a user asks Perplexity a question. Perplexity writes that because this fetch is initiated by the user, it generally does not apply robots.txt rules, and that this bot does not collect content for training. ### Google Google has no separate "AI search bot". Googlebot builds the Google Search index. According to Google's [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) document, for a page to be shown as a supporting link in Google AI Overviews or AI Mode, it must be indexed and eligible to be shown with a snippet in Google Search. Blocking Googlebot therefore means dropping out of both places at once. According to [Google's list of common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google-Extended is not a bot with a separate user agent string; crawling is done by Google's existing bots, and Google-Extended is used only as a control name in robots.txt. Google-Extended manages whether content Google crawls may be used to train future Gemini models, and whether it may be used for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. The same document writes that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. If you want to limit how much of your page is shown in AI Overviews, the route Google points to is not robots.txt but page-level snippet controls: `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex`. {{cta}} ## Block training, stay visible in search: a sample robots.txt The example below shows what a robots.txt looks like that allows search and user request bots and stops bots that collect for training. It is not a recommendation but an example of a decision that separates the roles; leaving training bots allowed is also a legitimate choice. ```text # Search index and user request bots: allowed User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Bots that collect for model training: blocked User-agent: GPTBot User-agent: ClaudeBot Disallow: / # All other crawlers (including Googlebot) User-agent: * Disallow: /yonetim/ Sitemap: https://www.ornek.com.tr/sitemap.xml ``` Note three things in this file. First, a group can carry more than one `User-agent` line; the rules beneath it apply to all of them. Second, Googlebot is not listed separately: it follows the wildcard group and crawls everywhere except the `/yonetim/` (admin) path. Third, the rule written for ChatGPT-User and Perplexity-User may not always be applied, according to the companies' own statements; even so, you have stated your intent clearly. > [!UYARI] > According to Google's documentation, adding Google-Extended to the training group stops your content from being used for Gemini training, but it also stops it from being used for grounding in Gemini Apps. It does not affect Google Search, AI Overviews or AI Mode. If being a source in answers in the Gemini app matters to you, weigh this line as a separate decision. Finally, the `Sitemap` line is independent of every group and can sit anywhere in the file. Open the sitemap in your browser to check that it really returns an XML file and does not show your site's design. ## Cloudflare's 2026 default Even if robots.txt allows it, a request can be stopped before it reaches your server. [On 1 July 2026, Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/) announced that it splits AI crawlers into three categories: Search (behaviour that collects or indexes content in order to answer questions later), Agent (automated behaviour acting on behalf of a person, usually doing a task in the moment) and Training (a crawler that takes content to train or fine-tune a model). According to the announcement, from 15 September 2026, for domains newly joining Cloudflare, crawlers in the Training and Agent categories are blocked by default on pages that show ads; the Search category remains allowed. Customers can choose to opt out of this default. Multi-purpose crawlers that combine search and training are assessed by all of their behaviours: a site that chooses to block training also blocks a multi-purpose crawler that also trains. This change does not cover every site; the announcement refers to newly joining domains and pages that show ads. Even so, its practical consequence is clear: a bot setting in a CDN dashboard can create a block that never shows up in robots.txt. Cloudflare's definition of Agent covers behaviour that runs in the moment on behalf of a person; this definition is close to bots that open a page at a user's request. Check in your own CDN dashboard which bot is counted in which category. ## How to check after writing the rule Whether a robots.txt rule works is understood not by looking at the file but by looking at the response the bot receives. We recommend four steps for checking: 1. **Open the file itself.** The address `https://yourdomain.com.tr/robots.txt` should show plain text. On single-page applications this address sometimes returns the site's home page with a 200 code; the file appears to exist, but it does not. 2. **Read the decision for each bot separately.** Check one by one which group applies to which bot, according to the matching rule above. 3. **Look at your CDN and firewall settings.** If a bot gets a 403 even though robots.txt allows it, the rule is most likely in front of the server. 4. **Read your server logs.** That is where the conclusive proof of whether a bot actually came lies. OpenAI and Perplexity publish their bots' IP addresses; not every request carrying a bot's name may come from that company. Our [scan, which requires no sign-up](/en/tarama), does part of these checks in one go: it reads your robots.txt rules separately for each AI bot, and also requests the same page under the AI bots' names to see whether the server treats these requests differently. The status of training bots is shown as a separate note that does not lower the score. How our own crawler identifies itself in your server logs is set out in our [crawler statement](/en/bot). For other technical signals such as page-level snippet controls and structured data, see our [structured data guide](/en/blog/yapisal-veri-schema-geo); for the current state of the llms.txt file, see our [llms.txt article](/en/blog/llms-txt-ise-yariyor-mu). ## Frequently asked questions ### If I block GPTBot, will I become invisible in ChatGPT? According to OpenAI's documentation, no. GPTBot is used for model training; the bot that determines being shown in ChatGPT's search answers is OAI-SearchBot. A site that blocks GPTBot and allows OAI-SearchBot remains eligible to be shown in ChatGPT search, according to OpenAI's statement. ### If I block Google-Extended, will I drop out of AI Overviews? According to Google's documentation, no. Google-Extended does not affect a site's inclusion in Google Search; to be a supporting link in AI Overviews and AI Mode, the page must be indexed in Google and eligible to be shown with a snippet. Google-Extended, for its part, manages Gemini training and use for grounding in Gemini Apps. ### Is robots.txt legally binding on AI companies? robots.txt is not access control but a request addressed to crawlers; RFC 9309, the protocol's standard document, says so explicitly. OpenAI, Anthropic, Perplexity and Google recommend in their own documentation that you manage their search and training bots with robots.txt; this is not a legal obligation but the companies' own statement. For a hard block, the rule needs to be written on the server or in the CDN. ### Does it make sense to block bots that come at a user's request? These bots open your page on the spot when a user asks a question or gives an address; if they are blocked, the engine cannot read your content at that moment. Because OpenAI and Perplexity write that robots.txt rules may not apply to these bots, the rule may not always work. If no data policy requires it, leaving them allowed lets your page be read at the moment the user asks. ### When does a change take effect? RFC 9309 says crawlers should generally not use a cached copy of robots.txt for more than 24 hours. OpenAI states about 24 hours for OAI-SearchBot, and Perplexity up to 24 hours. If you still see different behaviour after a day, look at your CDN settings and server logs. ## Sources - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), OpenAI, accessed 7 October 2026 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic Help Center, updated 7 April 2026 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots), Perplexity, accessed 7 October 2026 - [List of Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google Search Central, updated 14 July 2026 - [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), Google Search Central, updated 10 December 2025 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), IETF / RFC Editor, September 2022 - [Your site, your rules: new AI traffic options for all customers](https://blog.cloudflare.com/content-independence-day-ai-options/), Cloudflare Blog, 1 July 2026