Fix · AI access
Anthropic and Perplexity crawlers in robots.txt
Two more providers, five more tokens, and the same trap as OpenAI's: the crawler people block in a hurry is not the one that cites them.
The five, from each provider's own page
| Token | Provider | What it does | robots.txt |
|---|---|---|---|
ClaudeBot | Anthropic | Collects web content that may contribute to training. | Applies |
Claude-SearchBot | Anthropic | Improves "the relevance and accuracy of search responses" for Claude users. | Applies |
Claude-User | Anthropic | Fetches a page "when individuals ask questions to Claude". | Applies, per Anthropic |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; "not used for training". | Applies |
Perplexity-User | Perplexity | Visits a page when a user asks about it. "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." | Generally ignored |
Address lists: Anthropic publishes claude.com/crawling/bots.json; Perplexity
publishes perplexitybot.json and perplexity-user.json on its own domain, each with a
creation date. Read 22 September 2026.
Why it matters
A robots.txt line that names ClaudeBot opts you out of Anthropic training and changes
nothing about whether Claude can cite you: that is Claude-SearchBot. Likewise
PerplexityBot is the search crawler and does not train. The two user-triggered fetchers
behave like a person clicking a link; blocking them in robots.txt has no documented effect on
Perplexity-User and may not have one on Claude-User.
The repair
- Decide the two policies separately: training (ClaudeBot) and citation (Claude-SearchBot, PerplexityBot).
- Write only the named groups you need. Under RFC 9309, a bot with no named group follows
*. If*allows the site, the search crawlers are already allowed. - If you must block a fetcher rather than ask it, do it at the server or content delivery network (CDN), and only ever by user agent and the provider's published address list together. A rule on the user-agent string alone blocks anyone who copies the string and misses anyone who does not.
Before and after
Now
User-agent: *
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
# "Blocked the AI bots." Also blocked
# citation on two surfaces.After
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Disallow: /account/
Sitemap: https://yourbusiness.co.uk/sitemap.xml
# Claude-SearchBot, Claude-User,
# PerplexityBot and Perplexity-User follow *.State the policy in the file. A comment line saying which crawlers are allowed and why costs nothing and saves the next person from undoing it.
Prove it worked. For each of the five tokens, find the group that applies
(named, or *) and check what it allows. Then run the free check: it evaluates robots.txt
under RFC 9309 for every one of these tokens and names any it finds blocked, separating search
crawlers from training crawlers and noting where robots.txt may not apply.
What it will not do
Allowing a crawler makes a page eligible, not cited. And robots.txt cannot see past your edge: if a bot-protection setting challenges these crawlers, the file is never read. Check the hosting dashboard's bot settings, and its crawler traffic report, before assuming the file is in charge.
Sources: Anthropic, "Does Anthropic crawl data from the web…" (support.claude.com); Perplexity, "Perplexity crawlers" (docs.perplexity.ai/guides/bots); RFC 9309. Read 22 September 2026.