# The research behind AI-ready documentation

> Part of [Lensy by Perseverance AI](https://perseveranceai.com/). The index of every page for AI agents is at [https://perseveranceai.com/llms.txt](https://perseveranceai.com/llms.txt).

Last updated: 2026-11-10

The research behind what Lensy checks: how document structure affects AI retrieval, which AI crawlers sites block, and how little JSON-LD changes AI citations.

By Perseverance AI Published Mar 22, 2026 Updated Nov 10, 2026

## TL;DR

- Document structure directly affects AI retrieval quality. DeepRead showed 10.3% improvement and RAPTOR showed 20% improvement with structure-aware processing.
- ClaudeBot is blocked by 69% of websites and GPTBot by 62%. 71% of sites that block training bots also block search crawlers, removing themselves from AI results entirely.
- Structured data helps search engines understand a page, but controlled studies find JSON-LD barely changes AI citations. Heading hierarchies and canonical URLs matter more.

## Why document structure matters

Most AI search engines use Retrieval-Augmented Generation (RAG). They break documents into chunks, embed them in vector space, retrieve relevant chunks for a query, and feed those chunks to a language model. The quality of that chunking, and the structural signals available to guide it, directly affects answer quality.

DeepRead [1] found that preserving heading hierarchy during HTML-to-Markdown conversion enables a "locate-then-read" paradigm, improving retrieval by 10.3% over baseline agentic search. RAPTOR [2] showed 20% improvement on multi-step reasoning by building recursive tree structures from document hierarchies. The DUE benchmark [3] further established that document understanding requires explicit layout awareness across tasks spanning VQA, key information extraction, and machine reading comprehension.

Snowflake's engineering team [4] confirmed that Markdown-aware chunking provides 5–10% accuracy improvement in RAG quality for complex documents, and that retrieval strategy matters even with long-context LLMs.

## Bot access: the crawler gate

Major AI companies operate multiple crawlers with distinct purposes. Anthropic runs ClaudeBot (training), Claude-User (real-time fetches), and Claude-SearchBot (indexing). OpenAI separates GPTBot (training) from OAI-SearchBot (search results). Google separates Googlebot (search ranking) from Google-Extended (Gemini training) [7].

Industry analysis shows ClaudeBot is blocked by approximately 69% of websites and GPTBot by 62% [5]. More critically, 71% of sites that block training bots also inadvertently block search and retrieval bots [6], effectively removing themselves from AI-powered search results. OpenAI has stated that sites blocking OAI-SearchBot will not appear in ChatGPT search answers.

## Structured data

JSON-LD uses the Schema.org vocabulary to declare page types (`TechArticle`, `APIReference`), authors, publication dates, and topic relationships. Controlled data does not show it raising AI citations: Ahrefs [8] tracked 1,885 pages that added schema against 4,000 controls (May 2026), and their AI citations barely moved. That is why Lensy gives JSON-LD only 5 of the 100 AI search points. Google's Search Central documentation [9] confirms that structured data directly affects how content appears in search results and AI overviews.

OpenGraph meta tags influence how AI-powered preview systems and content aggregation tools summarise your content. Canonical URLs consolidate link equity and prevent AI engines from indexing duplicate versions [10], which is especially important for documentation sites serving versioned or localised content.

## What this means

84% of developers now use or plan to use AI tools [11]. Documentation that meets structural, access, and metadata standards gets cited. Documentation that does not is invisible to this growing majority.

## References

1. [1] [Li et al. "DeepRead: Document Structure-Aware Reasoning." arXiv:2602.05014, 2026.](https://arxiv.org/abs/2602.05014)
2. [2] [Sarthi et al. "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval." ICLR 2024.](https://arxiv.org/abs/2401.18059)
3. [3] [Borchmann et al. "DUE: End-to-End Document Understanding Benchmark." NeurIPS 2021.](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/069059b7ef840f0c74a814ec9237b6ec-Abstract-round2.html)
4. [4] [Snowflake. "How Retrieval & Chunking Impact Finance RAG." 2024.](https://www.snowflake.com/en/engineering-blog/impact-retrieval-chunking-finance-rag/)
5. [5] [ALM Corp. "ClaudeBot, Claude-User & Claude-SearchBot: Anthropic's Three-Bot Framework." 2025.](https://almcorp.com/blog/anthropic-claude-bots-robots-txt-strategy/)
6. [6] [Search Engine Journal. "Anthropic's Claude Bots Make Robots.txt Decisions More Granular." 2025.](https://www.searchenginejournal.com/anthropics-claude-bots-make-robots-txt-decisions-more-granular/568253/)
7. [7] [Google. "Google's Common Crawlers." Google for Developers, 2025.](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
8. [8] [Ahrefs. "We Tracked 1,885 Pages Adding Schema." 2026.](https://ahrefs.com/blog/schema-ai-citations/)
9. [9] [Google. "Introduction to Structured Data." Search Central.](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data)
10. [10] [Google. "Canonicalisation." Search Central.](https://developers.google.com/search/docs/crawling-indexing/canonicalization)
11. [11] [Stack Overflow. "2025 Developer Survey Results." 2025.](https://survey.stackoverflow.co/2025/)

Check your documentation's AI readiness.

[Start a free scan](https://perseveranceai.com/#scan)
