Free AI crawler access check
Can AI assistants fetch your website?
Assistants such as ChatGPT, Claude and Perplexity answer from pages their crawlers are allowed to fetch. This check reads your robots.txt and tells you whether those crawlers are turned away. Free, no signup, about a minute.
Confirming you are a person
Cloudflare runs this before each check. If a box appears, tick it. The card closes by itself.
What this check reads
Your robots.txt, against the crawlers that fetch pages so an assistant can answer with them and cite them:
- OAI-SearchBot and ChatGPT-User (OpenAI)
- Claude-SearchBot and Claude-User (Anthropic)
- PerplexityBot and Perplexity-User (Perplexity)
If any of them is disallowed, the check fails and names which ones.
Training crawlers are a choice, not a fault
GPTBot, ClaudeBot, CCBot and Applebot-Extended collect pages for training. Blocking them does not stop an assistant fetching your page to cite it, so the check reports it as a decision and does not fail you for it. Google-Extended is the exception worth knowing: it covers Gemini's grounding as well as training, so refusing it withdraws the page from what Gemini answers draw on.
What it cannot tell you
It reads the published rules only. Whether each provider actually reaches your site needs your server logs, and a firewall or content delivery network can block a crawler that robots.txt allows. ChatGPT-User and Perplexity-User fetch when a person asks about a page, and their providers document that robots.txt may not apply to those requests.
If it fails: the repair, step by step · Which OpenAI crawlers to allow · Anthropic and Perplexity crawlers
The answer leads; the full result follows
This page runs the whole EdenLens free check and puts the answer to its own question first. Underneath you get the rest of the result: up to three verified problems across the page, each with its repair.
Example results
- Wraycroft Creamery, before the repairs: three verified problems
- Wraycroft Creamery, after the repairs: no repair findings
- Eden Valley Joinery: insecure connection, blocked AI crawlers, opening hours missing
- Kinetiq Studio: content that only appears after JavaScript runs
All four are invented businesses. Each page is the engine's own output, published so it can be read without running a check.
The other free tools
Each one runs the same check and leads with its own question.
- The full free audit: up to three verified problems across all of it
- AI crawler access: can ChatGPT, Claude and Perplexity fetch the site?
- Business identity: can a machine tell who the business is?
- Sitemap: does a sitemap list the pages worth finding?
- Readable content: is the page readable before JavaScript runs?
What to do next
The paid reviews add wider investigation, human judgement and accountable advice. What this check could not assess is not a failure, and the free repairs stay free.