OpenAI’s crawlers at a glance
Everything below comes from OpenAI’s crawler documentation. OpenAI adds a version number to each user agent and says it may change, so match on the name (for example OAI-SearchBot), not the full string.
| User agent | What it does | What robots.txt controls | IP ranges |
|---|---|---|---|
OAI-SearchBot | Finds pages to show in ChatGPT search results. Not used for training. | Controls whether your pages can appear in ChatGPT search answers. | openai.com/searchbot.json |
GPTBot | Crawls content that may be used to train OpenAI’s foundation models. | Disallowing it says your content should not be used for training. | openai.com/gptbot.json |
ChatGPT-User | Fetches a page when a person asks ChatGPT or a Custom GPT to do something that needs it. Not automatic crawling. | Robots.txt rules may not apply, because the request comes from a user. Not used to decide search inclusion. | openai.com/chatgpt-user.json |
OAI-AdsBot | Checks landing pages submitted as ads on ChatGPT. Only visits those pages; not used for training. | OpenAI does not document a robots.txt control for this agent. | openai.com/adsbot.json |
GPTBot vs OAI-SearchBot: training vs search
GPTBot is about training: OpenAI uses it to crawl content that may be used to make its foundation models more useful and safe. OAI-SearchBot is about search: it is how pages get into the results ChatGPT shows when it searches the web.
OpenAI says each setting is independent of the others. A site can allow OAI-SearchBot so it appears in search results and disallow GPTBot to say its content should not be used for training. If a site allows both, OpenAI may use the results of one crawl for both purposes to avoid crawling twice.
The practical point: a site that blocked “OpenAI” in a hurry by disallowing every OpenAI user agent has also opted out of ChatGPT search. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links.
ChatGPT-User: requests a person asked for
ChatGPT-User is not a crawler in the usual sense. It makes a request when someone asks ChatGPT, or a Custom GPT, to do something that needs a web page. OpenAI says it is not used to crawl the web automatically and is not used to decide whether content can appear in search. Because a person starts these requests, OpenAI says robots.txt rules may not apply to them. To manage search and automatic crawling, use OAI-SearchBot.
OAI-AdsBot: ad landing pages only
OAI-AdsBot checks the safety of landing pages submitted as ads on ChatGPT. It only visits pages submitted as ads, and OpenAI says the data it collects is not used to train generative AI foundation models. If you don’t run ads on ChatGPT, you won’t see it.
robots.txt examples
Appear in ChatGPT search, opt out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Allow both search and training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /Opt out of both
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /Only do this if you don’t want to appear in ChatGPT search answers at all. OpenAI says search results can take about 24 hours to reflect a robots.txt change.
A general User-agent: * group with Disallow: / also applies to these crawlers unless they have their own group. If you block everything by default, add an explicit group for OAI-SearchBot when you want to appear in ChatGPT search.
Don’t forget firewalls and bot protection
Robots.txt is not the only thing that can keep OAI-SearchBot out. OpenAI recommends allowing requests from its published IP ranges, linked in the table above. If a firewall, CDN rule or bot-protection service challenges or blocks those addresses, the crawler may be unable to read your pages even when robots.txt allows it. Compare the connecting IP with the published ranges to verify requests that claim to come from an OpenAI crawler.
Check what your site allows today
The free AI visibility checker reads your robots.txt and reports which AI crawlers it allows or blocks, including OAI-SearchBot, GPTBot and ChatGPT-User, alongside which pages OpenAI’s API with web search cites for your buyers’ questions. No sign-up. For the next step after crawling, see how to get cited by ChatGPT.
OpenAI crawler FAQ
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot crawls pages so they can appear in ChatGPT search results. OpenAI says each robots.txt setting is independent, so you can allow one and block the other.
Does blocking GPTBot remove my site from ChatGPT search?
OpenAI's documentation treats them separately: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to say its content should not be used for training. Search inclusion is controlled with OAI-SearchBot.
What is ChatGPT-User?
ChatGPT-User makes requests when a person asks ChatGPT or a Custom GPT to do something that needs a web page. OpenAI says it is not used to crawl the web automatically and not used to decide whether content appears in search, and that robots.txt rules may not apply to these user-initiated requests.
How do I make sure my site can appear in ChatGPT search?
OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
How long does a robots.txt change take to apply?
For search results, OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.
What is OAI-AdsBot?
OAI-AdsBot checks the safety of landing pages submitted as ads on ChatGPT. It only visits pages submitted as ads, and OpenAI says the data it collects is not used to train foundation models.