BBox DocVQA: A Large Scale Bounding Box Grounded Dataset for Enhancing Reasoning in Document Visual Question Answer

TL;DR

BBox-DocVQA targets a core gap in DocVQA by enabling explicit bounding-box grounding of evidence. It combines a Segment-Judge-Generate pipeline with large multilingual VLMs to create a scalable, region-level QA dataset spanning single and multi-page documents. The dataset includes a large automatically generated training set and a carefully curated fine-grained benchmark to evaluate spatial grounding and reasoning. Experiments reveal that even strong VLMs struggle with precise grounding, underscoring the need for grounding-aware architectures and multi-step reasoning in document understanding.

Abstract

Document Visual Question Answering (DocVQA) is a fundamental task for multimodal document understanding and a key testbed for vision language reasoning. However, most existing DocVQA datasets are limited to the page level and lack fine grained spatial grounding, constraining the interpretability and reasoning capability of Vision Language Models (VLMs). To address this gap, we introduce BBox DocVQA a large scale, bounding box grounded dataset designed to enhance spatial reasoning and evidence localization in visual documents. We further present an automated construction pipeline, Segment Judge and Generate, which integrates a segment model for region segmentation, a VLM for semantic judgment, and another advanced VLM for question answer generation, followed by human verification for quality assurance. The resulting dataset contains 3.6 K diverse documents and 32 K QA pairs, encompassing single and multi region as well as single and multi page scenarios. Each QA instance is grounded on explicit bounding boxes, enabling fine grained evaluation of spatial semantic alignment. Benchmarking multiple state of the art VLMs (e.g., GPT 5, Qwen2.5 VL, and InternVL) on BBox DocVQA reveals persistent challenges in spatial grounding and reasoning accuracy. Furthermore, fine tuning on BBox DocVQA substantially improves both bounding box localization and answer generation, validating its effectiveness for enhancing the reasoning ability of VLMs. Our dataset and code will be publicly released to advance research on interpretable and spatially grounded vision language reasoning.

Figures (9)

• Figure 1: Comparison among (a) page--level data, (b) token--level data, (c) component--isolated data, and (d) our proposed BBox-DocVQA.
• Figure 2: An overview of the collected dataset and the proposed pipeline. We collect 4K PDFs across eight academic domains, segment page regions using SAM, filter and validate them with Qwen, and generate grounded QA pairs with GPT. We consider three annotation granularities—MPMBB, SPMBB, and SPSBB—covering multi-page, multi-region, and single-region reasoning.
• Figure 3: The left panels show the proportions of papers across research domains for (top) the document dataset and (bottom) the generated QA pairs. The right panels display the distributions of the number of figures, tables, and text blocks per paper.
• Figure 4: A case study demonstrating that the fine-grounded subimage can help the VLM to generate answers more accurately. The VLM is Qwen2.5VL-7B.
• Figure 5: Prompt used for automatic crop judging and classification.