Quick answer
Decide AI crawler access by purpose. Search and citation crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot affect whether AI tools can find and cite your pages. Training crawlers such as GPTBot, ClaudeBot and CCBot, and tokens like Google-Extended and Applebot-Extended, control use of your content for model training. Configure robots.txt per user agent, check CDN bot rules that may override it and review providers’ current documentation regularly.
Key takeaways
- Separate training crawlers from search and citation crawlers.
- robots.txt is a voluntary standard; reputable crawlers honour it.
- CDN and firewall bot rules can block crawlers regardless of robots.txt.
- Revisit choices as providers add or rename user agents.
Who this guide is for
Website owners, publishers and marketing teams deciding how to handle GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and similar crawlers.
Understand the types of AI crawlers
AI providers typically operate several user agents. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search results and ChatGPT-User for actions users ask ChatGPT to take. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User with similar distinctions. Perplexity documents PerplexityBot and Perplexity-User. Common Crawl’s CCBot builds a public dataset used by many researchers. Google-Extended and Applebot-Extended are robots.txt tokens controlling use of content crawled by Googlebot or Applebot for AI training and certain features, not separate crawlers.
Decide what you want
For most businesses seeking leads, being discoverable and citable in AI search is valuable, so allowing search-oriented crawlers makes sense. Training use is a separate question: some organizations allow it to increase the chance that models learn about them; others, particularly publishers and owners of proprietary content, block it. There is no universally right answer. Agree a policy with marketing, legal and leadership rather than leaving it to default settings.
Write clear robots.txt rules
robots.txt rules are grouped by user agent. For example, you might allow OAI-SearchBot and PerplexityBot to crawl the whole site while disallowing GPTBot and CCBot, and add a Google-Extended disallow if you do not want content used for Gemini training. Rules can also restrict specific directories, such as client portals or gated resources. Keep the file simple, comment your intent and test it with available robots.txt testing tools.
Remember robots.txt limitations
The Robots Exclusion Protocol, standardized as RFC 9309, is voluntary. Reputable crawlers respect it, but it is not access control. Content behind logins should be protected by authentication, not robots.txt. Also, blocking crawling does not necessarily remove information already collected, and user-initiated agents may fetch pages when a user provides a URL, under providers’ own policies.
Check CDN and firewall settings
Many CDNs and security services now offer one-click AI bot blocking or challenge pages. These can override a permissive robots.txt and silently prevent search-oriented AI crawlers from reaching your content. Review bot management settings, allow documented crawlers you want, verify them using providers’ published IP ranges where offered, and monitor server logs for blocked requests from those user agents.
Consider llms.txt and other signals
Some sites add an llms.txt file summarizing important content for language models. It is a community proposal, not a standard adopted by major search engines, and it does not replace robots.txt controls. Meta tags such as noai are not universally recognized either. Rely on documented controls from each provider and treat experimental signals as optional extras.
Review regularly
Providers add, rename and change crawlers and their purposes. Schedule a quarterly review of robots.txt, CDN settings and server logs. Track whether AI referral traffic and citations change after policy updates. Document the reasoning behind your choices so future teams understand the trade-offs rather than reversing them accidentally.
Scope checklist
- Which AI crawlers currently access your site, according to logs?
- What is your policy on training versus search use?
- Does robots.txt implement that policy per user agent?
- Do CDN or firewall rules block crawlers you intended to allow?
- Who reviews the policy and how often?
Frequently asked questions
Should I block GPTBot?
It depends on whether you want your content used for OpenAI model training. Blocking GPTBot is separate from allowing OAI-SearchBot, which supports ChatGPT search visibility.
Does blocking Google-Extended affect my Google rankings?
Google states that Google-Extended does not affect inclusion or ranking in Google Search; it controls use of content for Gemini training and certain grounding.
Is robots.txt legally binding?
robots.txt is a voluntary protocol rather than a security or legal mechanism. Protect private content with authentication and consult legal advice on content use rights.
Sources and implementation references
- OpenAI: overview of OpenAI crawlers
GPTBot, OAI-SearchBot and ChatGPT-User.
- Google Search Central: Google’s common crawlers
Googlebot and the Google-Extended token.
- RFC 9309: Robots Exclusion Protocol
The standardized robots.txt protocol.
Prepared with AI assistance as an engineering decision guide. Examples are illustrative unless explicitly labelled as an internal implementation. No independent expert review or customer outcome is claimed.
Want more qualified enquiries from search and AI answers?
Share your website, main services and target markets. We review technical, content and conversion gaps before proposing scope. Scope, access, dependencies and commercial terms are agreed before delivery. Do not submit passwords, API keys or confidential records.