Fix · Access

AI assistants cannot fetch your pages

Your robots.txt refuses a crawler that assistants use to fetch pages at the moment they answer. Content quality cannot rescue a page that cannot be retrieved.

What we found

A2 · Retrieval crawler blocked — robots.txt blocks OAI-SearchBot. Assistants that respect these rules cannot fetch this site to cite it, whatever the content says. Evidence: robots.txt names the crawler

First, the distinction nobody else makes

There are two kinds of AI crawler and they do completely different jobs. Most tools report them as one thing, which is why businesses are told they are invisible when they have simply made a choice about training.

Purpose	Agents	Blocking it means
Retrieval fetches your page so an assistant can answer and cite	OAI-SearchBot · ChatGPT-User · Claude-SearchBot · Claude-User · PerplexityBot · Perplexity-User	You are removed from answers. This is the gate.
Training collects content for model training	GPTBot · ClaudeBot · CCBot · Applebot-Extended	Nothing, for citation. A rights decision, not a fault.
Google-Extended a robots.txt token, not a crawler	Google-Extended	Out of Gemini's answers. It covers training and Gemini grounding together; Google Search itself is unaffected.

Roles as each provider states them, re-read 21 September 2026: OpenAI's crawler page, Anthropic's crawler help article and Perplexity's crawler page. ChatGPT-User and Perplexity-User fetch only when a person asks about a page, and both providers say robots.txt may not apply to those requests.

If you deliberately block GPTBot because you do not want your writing training a model, keep doing it. That is a legitimate choice about your own work and it does not stop anyone citing you. We report it as a note, never as a failure.

The repair, in four steps

Open robots.txt on every host you run. It sits at the root: https://example.co.uk/robots.txt. A subdomain needs its own file.
Find every Disallow: / that reaches a retrieval agent. Those are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Look under their own names, and under User-agent: * if they are not named anywhere: a crawler with no group of its own obeys the * group. Several user-agent lines stacked above one rule all share it.
Remove the retrieval block, and keep any training refusal you meant. Use the "after" block below. You do not need a group that says Allow: / for each retrieval agent: one that is not named follows *, and naming it replaces the * rules for that crawler entirely, so your /cart/ and /account/ exclusions would stop applying to it. Refusing GPTBot, ClaudeBot or CCBot is a separate decision about your writing, and this repair does not reverse it.
Load /robots.txt in a browser and read it back. It must answer HTTP 200 with plain text. A 404 is harmless (crawlers treat it as "no rules"). A server error is not: under the robots standard, RFC 9309, a crawler that gets a 5xx must treat the whole site as off limits until the file answers again.

Before and after

Now — blocks retrieval

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Disallow: /          # removes you from
                       # ChatGPT answers

After — retrieval allowed, training refused

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /account/

# Retrieval agents are not named, so they follow
# the * group above: allowed, with /cart/ and
# /account/ still excluded.

# Training — your call. This refuses it.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Google-Extended also removes you from Gemini's
# answers. Add it only if that is what you want.

Sitemap: https://example.co.uk/sitemap.xml

Check the page level too. robots.txt is not the only lever. A noindex in a meta tag or an X-Robots-Tag response header will keep you out of search regardless of what robots.txt permits, and those are easy to leave behind after a site rebuild.

Where it goes

robots.txt lives at the root of every host you run: https://example.co.uk/robots.txt. A subdomain needs its own. On Shopify and Wix the file is generated for you and edited through the platform rather than uploaded.

Prove it worked. Load /robots.txt in a browser and confirm it returns plain text, not a 404 page styled like your site. Then check that no Disallow: / sits under any retrieval agent. Expected result in your next EdenLens check: the AI access foundation passes, with a note recording the training crawlers you refused.

What it will not do

Allowing a crawler permits it; it does not summon it. Nothing here makes an assistant visit, and nothing here makes it choose you once it has. It removes a wall you did not know you had built.