Fix · AI access

GPTBot, OAI-SearchBot and ChatGPT-User: which to allow

OpenAI runs three crawlers with three different jobs. Most sites that "blocked ChatGPT" blocked the one that trains models and, by accident, the one that cites websites in answers. You can allow one without the other.

The three, from OpenAI's own documentation

Token	What it does	Does robots.txt apply?
`GPTBot`	Crawls content "that may be used in training" generative models. Disallowing it "indicates a site's content should not be used in training".	Yes
`OAI-SearchBot`	"Used to surface websites in search results in ChatGPT's search features." Sites that opt out "will not be shown in ChatGPT search answers".	Yes
`ChatGPT-User`	Fetches a page when a user asks ChatGPT about it. "Because these actions are initiated by a user, robots.txt rules may not apply."	May not

Each publishes its addresses: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json. Read 22 September 2026.

Why it matters

Allowing OAI-SearchBot is what makes your pages eligible to be cited in ChatGPT's search answers. OpenAI's documentation does not say that allowing it means your content is used for training; that is GPTBot's job, and you decide it separately. A single Disallow: / under User-agent: * removes you from both, and from every other engine, at once.

How robots.txt actually reads these (RFC 9309)

A named group wins for that bot. If there is a User-agent: GPTBot group, GPTBot follows it and ignores the * group entirely. The other two, unnamed, follow *.
Matching is exact and case-insensitive. "OAI-SearchBot" is one token; "OpenAI" or "ChatGPT" matches nothing.
You do not need a named group to allow a bot. If * already allows the site, OAI-SearchBot is already allowed. Add a named group only where you want a different rule.

Before and after

Now: "we blocked ChatGPT"

User-agent: *
Disallow: /

# Blocks Google, Bing, every AI crawler,
# and does not stop ChatGPT-User.

After: no training, yes citation

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /
Disallow: /account/
Disallow: /cart/

Sitemap: https://yourbusiness.co.uk/sitemap.xml
# OAI-SearchBot and ChatGPT-User follow *.

Both decisions are legitimate. Allowing training is a policy choice some businesses make deliberately; opting out is too. The defect is only ever blocking the search crawler by accident.

Prove it worked. Open /robots.txt. Find the group that applies to OAI-SearchBot (its own, or *) and confirm it does not disallow the pages you want cited. Then run the free check: the foundation "AI assistants are allowed to read the site" passes and, if GPTBot is disallowed, notes it as a documented training choice rather than a fault.

What it will not do

robots.txt is a request that well-behaved crawlers honour; it is not access control. Allowing OAI-SearchBot makes you eligible, it does not make you cited: that depends on whether your pages answer the question asked. And if your hosting or firewall blocks these crawlers at the edge, robots.txt never gets a say; that is a separate check.

Source: OpenAI, "Overview of OpenAI crawlers" (developers.openai.com/api/docs/bots), read 22 September 2026. RFC 9309, Robots Exclusion Protocol.