Crawler policy

EdenLensBot, and how edenlens.com treats crawlers.

What EdenLensBot is.

When someone runs the free check on edenlens.com, or a full review in an EdenLens app, EdenLens fetches a small number of public pages from the website being checked. Those requests identify themselves with this user agent:

EdenLensBot/0.2 (+https://edenlens.com/bot; free quick check)

The free check fetches at most the homepage, robots.txt and sitemap.xml, once, and reads no more than two megabytes of each. A full review fetches a bounded set of pages, by default about twelve, at a polite pace. EdenLensBot does not run JavaScript, does not submit forms, does not log in, and does not fetch anything a person did not ask it to check.

How to opt out.

Add a rule for the agent name to your robots.txt. EdenLensBot reads robots.txt on every check and reports a block as a documented choice, not a fault:

User-agent: EdenLensBot
Disallow: /

If a site blocks EdenLensBot, the free check says so and stops. If you believe EdenLensBot has behaved badly towards your site, email steven@integratedsynergyconsulting.com with the hostname and the time, and it will be looked at.

How edenlens.com treats crawlers.

EdenLens sells the idea that a business should decide, in the open, who may read its pages. So this site's own policy is published rather than implied. edenlens.com allows search crawlers and the crawlers that fetch pages for AI answers. It also allows the training crawlers, as a deliberate choice that can be revisited. Nothing is blocked at the edge, and the robots.txt is authored by hand, not generated:

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://edenlens.com/sitemap.xml

The /api/ paths are the free check and the waitlist, which have nothing to index.