Fix · Access

AI assistants cannot fetch your pages

Your robots.txt refuses a crawler that assistants use to fetch pages at the moment they answer. Content quality cannot rescue a page that cannot be retrieved.

What we found

A2 · Retrieval crawler blocked — robots.txt blocks OAI-SearchBot. Assistants that respect these rules cannot fetch this site to cite it, whatever the content says. Evidence EV-026

First, the distinction nobody else makes

There are two kinds of AI crawler and they do completely different jobs. Most tools report them as one thing, which is why businesses are told they are invisible when they have simply made a choice about training.

Purpose	Agents	Blocking it means
Retrieval fetches your page so an assistant can answer and cite	OAI-SearchBot · ChatGPT-User · Claude-SearchBot · Claude-User · PerplexityBot	You are removed from answers. This is the gate.
Training collects content for model training and grounding	GPTBot · Google-Extended · CCBot	Nothing, for citation. A rights decision, not a fault.

If you deliberately block GPTBot because you do not want your writing training a model, keep doing it. That is a legitimate choice about your own work and it does not stop anyone citing you. We report it as a note, never as a failure.

Before and after

Now — blocks retrieval

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Disallow: /          # removes you from
                       # ChatGPT answers

After — retrieval allowed, training refused

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /account/

# Retrieval — allowed, so you can be cited
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# Training — your call. This refuses it.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /

Sitemap: https://example.co.uk/sitemap.xml

Check the page level too. robots.txt is not the only lever. A noindex in a meta tag or an X-Robots-Tag response header will keep you out of search regardless of what robots.txt permits, and those are easy to leave behind after a site rebuild.

Where it goes

robots.txt lives at the root of every host you run: https://example.co.uk/robots.txt. A subdomain needs its own. On Shopify and Wix the file is generated for you and edited through the platform rather than uploaded.

Prove it worked. Load /robots.txt in a browser and confirm it returns plain text, not a 404 page styled like your site. Then check that no Disallow: / sits under any retrieval agent. Expected result in your next EdenLens check: the AI access foundation passes, with a note recording the training crawlers you refused.

What it will not do

Allowing a crawler permits it; it does not summon it. Nothing here makes an assistant visit, and nothing here makes it choose you once it has. It removes a wall you did not know you had built.