AI Readiness/3 min read

The research behind AI-ready documentation

The research behind what Lensy checks: how document structure affects AI retrieval, which AI crawlers sites block, and how little JSON-LD changes AI citations.

By Perseverance AI Published Updated

TL;DR

  • Document structure directly affects AI retrieval quality. DeepRead showed 10.3% improvement and RAPTOR showed 20% improvement with structure-aware processing.
  • ClaudeBot is blocked by 69% of websites and GPTBot by 62%. 71% of sites that block training bots also block search crawlers, removing themselves from AI results entirely.
  • Structured data helps search engines understand a page, but controlled studies find JSON-LD barely changes AI citations. Heading hierarchies and canonical URLs matter more.

Why document structure matters

Most AI search engines use Retrieval-Augmented Generation (RAG). They break documents into chunks, embed them in vector space, retrieve relevant chunks for a query, and feed those chunks to a language model. The quality of that chunking, and the structural signals available to guide it, directly affects answer quality.

DeepRead [1] found that preserving heading hierarchy during HTML-to-Markdown conversion enables a "locate-then-read" paradigm, improving retrieval by 10.3% over baseline agentic search. RAPTOR [2] showed 20% improvement on multi-step reasoning by building recursive tree structures from document hierarchies. The DUE benchmark [3] further established that document understanding requires explicit layout awareness across tasks spanning VQA, key information extraction, and machine reading comprehension.

Snowflake's engineering team [4] confirmed that Markdown-aware chunking provides 5–10% accuracy improvement in RAG quality for complex documents, and that retrieval strategy matters even with long-context LLMs.

Bot access: the crawler gate

Major AI companies operate multiple crawlers with distinct purposes. Anthropic runs ClaudeBot (training), Claude-User (real-time fetches), and Claude-SearchBot (indexing). OpenAI separates GPTBot (training) from OAI-SearchBot (search results). Google separates Googlebot (search ranking) from Google-Extended (Gemini training) [7].

Industry analysis shows ClaudeBot is blocked by approximately 69% of websites and GPTBot by 62% [5]. More critically, 71% of sites that block training bots also inadvertently block search and retrieval bots [6], effectively removing themselves from AI-powered search results. OpenAI has stated that sites blocking OAI-SearchBot will not appear in ChatGPT search answers.

Structured data

JSON-LD uses the Schema.org vocabulary to declare page types (TechArticle, APIReference), authors, publication dates, and topic relationships. Controlled data does not show it raising AI citations: Ahrefs [8] tracked 1,885 pages that added schema against 4,000 controls (May 2026), and their AI citations barely moved. That is why Lensy gives JSON-LD only 5 of the 100 AI search points. Google's Search Central documentation [9] confirms that structured data directly affects how content appears in search results and AI overviews.

OpenGraph meta tags influence how AI-powered preview systems and content aggregation tools summarise your content. Canonical URLs consolidate link equity and prevent AI engines from indexing duplicate versions [10], which is especially important for documentation sites serving versioned or localised content.

What this means

84% of developers now use or plan to use AI tools [11]. Documentation that meets structural, access, and metadata standards gets cited. Documentation that does not is invisible to this growing majority.

Check your documentation's AI readiness.

Start a free scan