What it is
Lensy checks whether a documentation page is ready for AI search engines and coding agents. When someone enters a URL on perseveranceai.com, Lensy fetches that page and a few public files on the same site, such as robots.txt, llms.txt, the sitemap and the page's Markdown version. That's usually 10 to 25 requests per scan. It does not crawl your site on its own; each visit starts from a person's scan.
How to recognise it
Lensy names itself in every request:
Lensy/1.0 (+https://perseveranceai.com/bot)
Every scan comes from this IP address:
Loading…
We'll announce any change to this address on this page at least 14 days ahead.
If the site refuses the first request with HTTP 403 or 429, Lensy tries once more with a browser user agent that ends with the same (+https://perseveranceai.com/bot) link, from the same IP. If that is refused too, the scan stops and tells the person that the site's bot protection blocked the checker.
Lensy never sends the names of other companies' crawlers, such as GPTBot or ClaudeBot.
Command-line tool
The open-source Lensy command-line tool sends lensy-cli/<version> (+https://perseveranceai.com/bot), with the version of the tool that is running. It fetches only the pages the person running it asked for, plus the files those checks need: robots.txt, the sitemap, llms.txt and the page's Markdown version. The coding agent checks from AFDocs go out under the same name.
It runs on that person's own machine or CI runner, so its requests come from their address and not from the one above. Unlike a scan from perseveranceai.com, it never retries with a browser user agent. If a site refuses it, it stops and reports that the page wasn't checked.
A rule on the user agent alone can be copied by anyone. To let your own CI run the tool, allow that runner's IP address together with the user agent.
robots.txt
A Lensy scan is a fetch a person asked for, like an AI assistant opening a page on a user's behalf, so Lensy doesn't apply your robots.txt rules to itself. It reads robots.txt to report what your rules tell AI bots. To stop Lensy visiting, use one of the options under “How to block it”.
How to allow it
Match both the user agent and the IP address. A rule on the user agent alone lets in anyone who copies the name.
Cloudflare Pro and above: add a custom WAF rule with the action Skip for this expression, and choose to skip Super Bot Fight Mode, the managed rules, Browser Integrity Check and Security Level:
(http.user_agent contains "Lensy/1.0") and (ip.src in {<Lensy IP>})
Cloudflare free plan: Bot Fight Mode can't be skipped by a custom rule. Instead, add an IP Access rule with the action Allow for the IP above; Bot Fight Mode doesn't act on requests an IP Access rule allows. This matches the IP only, which is reasonable here because that address is dedicated to Lensy and used by nothing else.
Other firewalls and CDNs have an equivalent “allow when user agent and source IP match” rule.
You don't have to do this. A blocked scan doesn't mean verified AI crawlers are blocked, and we're working on getting Lensy onto Cloudflare's verified-bot list so it works without any change on your side.
How to block it
Block requests that match the user agent and IP above in your firewall, or email us and we'll stop scans of your domain.
Contact
Questions, or want Lensy to stop visiting your site? Email hello@perseveranceai.com.