OAI-SearchBot vs GPTBot — the key difference
OpenAI operates two separate crawlers with different purposes:
OAI-SearchBot is used for search discovery. It surfaces websites in ChatGPT Search results and cites them as sources in AI answers.
GPTBot is used for model training. It is associated with potential access to content for training OpenAI language models.
This distinction matters strategically. Allowing your content to appear in ChatGPT Search and allowing your content to be used for model training are two independent decisions — controlled by separate entries in robots.txt.
How to configure robots.txt
To allow OAI-SearchBot while blocking GPTBot:
User-agent: OAI-SearchBot Allow: /
User-agent: GPTBot Disallow: /
To allow both crawlers:
User-agent: OAI-SearchBot Allow: /
User-agent: GPTBot Allow: /
How to verify your current settings
Open your robots.txt file at yourdomain.com/robots.txt and check whether OAI-SearchBot is listed under any Disallow rule. If it is not mentioned at all, it is allowed by default.
Source
OpenAI documents OAI-SearchBot and GPTBot as independent crawlers in official documentation: platform.openai.com/docs/bots