LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding

TL;DR

Question answering over visually rich documents (VRDs) suffers from loss of layout context, overreliance on purely neural embeddings, and static retrieval strategies. LAD-RAG introduces a layout-aware dynamic retrieval framework that builds a symbolic document graph during ingestion and a neural index, then uses an LLM agent to adaptively retrieve evidence from both indices at inference time. The approach yields over 90% perfect recall on average without tuning, improves recall compared with strong baselines, and delivers significant end-to-end QA gains across four VRD benchmarks with minimal latency. This work demonstrates the value of jointly modeling document structure and content to enable complete, context-aware evidence gathering in complex, multi-page documents, with broad applicability to enterprise and scientific domains.

Abstract

Question answering over visually rich documents (VRDs) requires reasoning not only over isolated content but also over documents' structural organization and cross-page dependencies. However, conventional retrieval-augmented generation (RAG) methods encode content in isolated chunks during ingestion, losing structural and cross-page dependencies, and retrieve a fixed number of pages at inference, regardless of the specific demands of the question or context. This often results in incomplete evidence retrieval and degraded answer quality for multi-page reasoning tasks. To address these limitations, we propose LAD-RAG, a novel Layout-Aware Dynamic RAG framework. During ingestion, LAD-RAG constructs a symbolic document graph that captures layout structure and cross-page dependencies, adding it alongside standard neural embeddings to yield a more holistic representation of the document. During inference, an LLM agent dynamically interacts with the neural and symbolic indices to adaptively retrieve the necessary evidence based on the query. Experiments on MMLongBench-Doc, LongDocURL, DUDE, and MP-DocVQA demonstrate that LAD-RAG improves retrieval, achieving over 90% perfect recall on average without any top-k tuning, and outperforming baseline retrievers by up to 20% in recall at comparable noise levels, yielding higher QA accuracy with minimal latency.

Figures (12)

• Figure 1: LAD-RAG addresses three key limitations of conventional RAGs in VRDs by (1) introducing a symbolic document graph to capture layout and cross-page structure, (2) integrating symbolic and neural indices to preserve structural and semantic signals, and (3) leveraging an LLM agent for dynamic, query-adaptive retrieval beyond static top‑$k$ methods.
• Figure 2: LAD-RAG framework: During ingestion, an LVLM extracts elements from each page and encodes them into an index, while also constructing a document graph that captures inter- and intra-page relationships for symbolic search. During inference, an agent interprets the question and iteratively interacts with both the neural index and the document graph to retrieve relevant evidence, enabling question-aware, structure-aware retrieval.
• Figure 3: Retrieval performance of LAD-RAG compared to baseline retrievers across varying top-$k$ settings. Baselines retrieve from $k=1$ up to the point of perfect recall. Dotted horizontal lines indicate the number of retrieved pages each baseline requires to match the recall achieved by LAD-RAG without any top-$k$ tuning.
• Figure 4: Distribution of the number of LLM calls per query by LAD-RAG.
• Figure 5: Case study showing LAD-RAG retrieving all charts that compare the general public with Latinos/Hispanics. While a conventional semantic retriever fails to recall many relevant charts and includes irrelevant ones, LAD-RAG dynamically opts for the symbolic retrieval. It filters for all figure nodes and then contextualizes them using the surrounding layout to determine whether they match the query. This multi-step, graph-guided process enables accurate and exhaustive evidence collection.