Crawler policy

EdenLensBot, and how edenlens.com treats crawlers.

What EdenLensBot is.

When someone runs the free check on edenlens.com, or a full review in an EdenLens app, EdenLens fetches a small number of public pages from the website being checked. Those requests identify themselves with this user agent:

EdenLensBot/0.2 (+https://edenlens.com/bot; free quick check)

The free check reads robots.txt first, before any page. If that allows it, it then fetches the homepage over https, asks the plain http address once without following it (to see whether it redirects), and fetches sitemap.xml. If the homepage states no review figures but links the site’s own reviews page, it reads that one page too. It then opens up to eight profile pages that the homepage itself links to or declares, such as a Facebook or LinkedIn page, to check that they name the same business. For an Instagram account it reads the account’s public embed page, the one Instagram provides for showing a profile on other websites, because the profile page itself asks visitors to sign in. It also requests, once, the picture the homepage names for sharing. Each request is made once and no more than two megabytes of any response is read. A full review fetches a bounded set of pages, by default about twelve, at a polite pace. EdenLensBot does not run JavaScript, does not submit forms, does not log in, and does not fetch anything beyond the site a person asked it to check, the profiles that site points to, and the pictures those pages name.

How to opt out.

Add a rule for the agent name to your robots.txt. EdenLensBot reads robots.txt before anything else on every check, to the robots standard (RFC 9309), so a group that names it, alone or stacked with other agents, is honoured:

User-agent: EdenLensBot
Disallow: /

If a site blocks EdenLensBot, no page is requested: the free check says so and stops after reading robots.txt. If robots.txt answers with a server error, the standard says a crawler must treat the whole site as off limits, so EdenLensBot stops there too and reports the error, which is itself the thing to fix. If you believe EdenLensBot has behaved badly towards your site, email info@edenlens.com with the hostname and the time, and it will be looked at.

How edenlens.com treats crawlers.

EdenLens sells the idea that a business should decide, in the open, who may read its pages. So this site's own policy is published rather than implied. edenlens.com allows search crawlers and the crawlers that fetch pages for AI answers. It also allows the training crawlers, as a deliberate choice that can be revisited. Nothing is blocked at the edge, and the robots.txt is authored by hand, not generated:

# edenlens.com crawler policy, published in full at https://edenlens.com/bot
# Search crawlers, AI answer crawlers and training crawlers are all allowed.
# Training is allowed deliberately and can be revisited. /api/ has nothing to index.
User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://edenlens.com/sitemap.xml

The /api/ paths are the free check and the waitlist, which have nothing to index.