The research behind AI-ready documentation
The research behind what Lensy checks: how document structure affects AI retrieval, which AI crawlers sites block, and how little JSON-LD changes AI citations.
By Perseverance AI Published Updated
TL;DR
- Document structure directly affects AI retrieval quality. DeepRead showed 10.3% improvement and RAPTOR showed 20% improvement with structure-aware processing.
- ClaudeBot is blocked by 69% of websites and GPTBot by 62%. 71% of sites that block training bots also block search crawlers, removing themselves from AI results entirely.
- Structured data helps search engines understand a page, but controlled studies find JSON-LD barely changes AI citations. Heading hierarchies and canonical URLs matter more.
Why document structure matters
Most AI search engines use Retrieval-Augmented Generation (RAG). They break documents into chunks, embed them in vector space, retrieve relevant chunks for a query, and feed those chunks to a language model. The quality of that chunking, and the structural signals available to guide it, directly affects answer quality.
DeepRead [1] found that preserving heading hierarchy during HTML-to-Markdown conversion enables a "locate-then-read" paradigm, improving retrieval by 10.3% over baseline agentic search. RAPTOR [2] showed 20% improvement on multi-step reasoning by building recursive tree structures from document hierarchies. The DUE benchmark [3] further established that document understanding requires explicit layout awareness across tasks spanning VQA, key information extraction, and machine reading comprehension.
Snowflake's engineering team [4] confirmed that Markdown-aware chunking provides 5–10% accuracy improvement in RAG quality for complex documents, and that retrieval strategy matters even with long-context LLMs.
Bot access: the crawler gate
Major AI companies operate multiple crawlers with distinct purposes. Anthropic runs ClaudeBot (training), Claude-User (real-time fetches), and Claude-SearchBot (indexing). OpenAI separates GPTBot (training) from OAI-SearchBot (search results). Google separates Googlebot (search ranking) from Google-Extended (Gemini training) [7].
Industry analysis shows ClaudeBot is blocked by approximately 69% of websites and GPTBot by 62% [5]. More critically, 71% of sites that block training bots also inadvertently block search and retrieval bots [6], effectively removing themselves from AI-powered search results. OpenAI has stated that sites blocking OAI-SearchBot will not appear in ChatGPT search answers.
Structured data
JSON-LD uses the Schema.org vocabulary to declare page types (TechArticle, APIReference), authors, publication dates, and topic relationships. Controlled data does not show it raising AI citations: Ahrefs [8] tracked 1,885 pages that added schema against 4,000 controls (May 2026), and their AI citations barely moved. That is why Lensy gives JSON-LD only 5 of the 100 AI search points. Google's Search Central documentation [9] confirms that structured data directly affects how content appears in search results and AI overviews.
OpenGraph meta tags influence how AI-powered preview systems and content aggregation tools summarise your content. Canonical URLs consolidate link equity and prevent AI engines from indexing duplicate versions [10], which is especially important for documentation sites serving versioned or localised content.
What this means
84% of developers now use or plan to use AI tools [11]. Documentation that meets structural, access, and metadata standards gets cited. Documentation that does not is invisible to this growing majority.
References
- [1] Li et al. "DeepRead: Document Structure-Aware Reasoning." arXiv:2602.05014, 2026.
- [2] Sarthi et al. "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval." ICLR 2024.
- [3] Borchmann et al. "DUE: End-to-End Document Understanding Benchmark." NeurIPS 2021.
- [4] Snowflake. "How Retrieval & Chunking Impact Finance RAG." 2024.
- [5] ALM Corp. "ClaudeBot, Claude-User & Claude-SearchBot: Anthropic's Three-Bot Framework." 2025.
- [6] Search Engine Journal. "Anthropic's Claude Bots Make Robots.txt Decisions More Granular." 2025.
- [7] Google. "Google's Common Crawlers." Google for Developers, 2025.
- [8] Ahrefs. "We Tracked 1,885 Pages Adding Schema." 2026.
- [9] Google. "Introduction to Structured Data." Search Central.
- [10] Google. "Canonicalisation." Search Central.
- [11] Stack Overflow. "2025 Developer Survey Results." 2025.
Check your documentation's AI readiness.
Start a free scan