GPTBot vs OAI-SearchBot — the critical distinction
OpenAI operates two separate crawlers with fundamentally different purposes:
GPTBot is associated with collecting web content for potential use in training OpenAI language models. Allowing GPTBot gives OpenAI permission to use your content as training data.
OAI-SearchBot is used for search-related discovery and citation in ChatGPT Search features. Allowing OAI-SearchBot enables your website to appear as a cited source in ChatGPT Search results.
These are independent settings in robots.txt. A decision about one does not affect the other.
How to configure robots.txt for GPTBot
To block GPTBot while allowing OAI-SearchBot for ChatGPT Search visibility:
User-agent: GPTBot Disallow: /
User-agent: OAI-SearchBot Allow: /
To allow both:
User-agent: GPTBot Allow: /
User-agent: OAI-SearchBot Allow: /
To block both:
User-agent: GPTBot Disallow: /
User-agent: OAI-SearchBot Disallow: /
Should you block GPTBot?
This is a strategic decision, not a technical one. Considerations include:
- Whether your organization has a policy on AI training data use
- Whether you want your content to contribute to future model improvements
- Whether blocking training access while maintaining search visibility is the right balance
There is no universally correct answer. Many publishers block GPTBot as a precaution while keeping OAI-SearchBot allowed to maintain ChatGPT Search visibility.
How to verify your current robots.txt
Open yourdomain.com/robots.txt and check whether GPTBot appears under any User-agent directive. If it is not mentioned, it is allowed by default.
Source
OpenAI documents GPTBot and OAI-SearchBot as independent crawlers at platform.openai.com/docs/bots