AVIR: Adaptive Visual In-Document Retrieval for Efficient Multi-Page Document Question Answering

TL;DR

The paper tackles MP‑VQA by addressing the dual challenges of high computation and context dilution in large vision‑language models. It introduces AVIR, a retrieval‑augmented pipeline that uses a lightweight Pix2Struct‑based page retriever, an adaptive page selector with thresholding and clustering, and a frozen Qwen2.5‑VL reader to generate answers from a compact set of pages. Key contributions include (i) a systematic analysis of how irrelevant pages impair LVLMs, (ii) an adaptive page selection mechanism outperforming rigid Top‑K baselines, and (iii) a complete retrieval‑guided MP‑VQA pipeline that achieves $ANLS=0.8458$ on MP‑DocVQA and strong results on SlideVQA and DUDE while reducing pages processed by about $70\%$. The approach yields practical impact by enabling faster, scalable multi‑page document question answering with minimal fine‑tuning, though it trades full document coverage and some universal answer formats for efficiency.

Abstract

Multi-page Document Visual Question Answering (MP-DocVQA) remains challenging because long documents not only strain computational resources but also reduce the effectiveness of the attention mechanism in large vision-language models (LVLMs). We tackle these issues with an Adaptive Visual In-document Retrieval (AVIR) framework. A lightweight retrieval model first scores each page for question relevance. Pages are then clustered according to the score distribution to adaptively select relevant content. The clustered pages are screened again by Top-K to keep the context compact. However, for short documents, clustering reliability decreases, so we use a relevance probability threshold to select pages. The selected pages alone are fed to a frozen LVLM for answer generation, eliminating the need for model fine-tuning. The proposed AVIR framework reduces the average page count required for question answering by 70%, while achieving an ANLS of 84.58% on the MP-DocVQA dataset-surpassing previous methods with significantly lower computational cost. The effectiveness of the proposed AVIR is also verified on the SlideVQA and DUDE benchmarks. The code is available at https://github.com/Li-yachuan/AVIR.

Figures (3)

• Figure 1: The influence of document length on the accuracy of MP-VQA. The experiment is conducted on the validation set of MP-DocVQA, where the answers are usually on a single page. 'w/ doc' indicates using the entire document, while 'w/ true page' means only the correct page is used. Due to the uneven distribution of page lengths, we have only selected the first 202 documents for each category.
• Figure 2: Overview of the proposed AVIR. $P(x_0)$ means The relevance probability assigned by the retrieval model to the most relevant page.
• Figure 3: Four typical errors of AVIR on the DUDE dataset.