LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

TL;DR

The paper tackles the need for a comprehensive benchmark to evaluate long-document understanding by LVLMs across understanding, numerical reasoning, and cross-element locating. It introduces LongDocURL, a 20-sub-task benchmark built from 396 long documents (50–150 pages) and 2,325 QA pairs generated via a semi-automated pipeline, including evidence-grounded QA across text, layout, figures, and tables. Through 26 model configurations (open- and closed-source) and both image- and text-input paradigms, the study reveals substantial performance gaps, with GPT-4o achieving the top score (~64.5) while open-source models lag behind (~30.6), largely due to loss of structural information when text is extracted. Detailed error analysis shows perceptual and reasoning errors as major failure modes, highlighting the importance of robust layout parsing and multi-stage QA frameworks for advancing long-document understanding.

Abstract

Large vision language models (LVLMs) have improved the document understanding capabilities remarkably, enabling the handling of complex document elements, longer contexts, and a wider range of tasks. However, existing document understanding benchmarks have been limited to handling only a small number of pages and fail to provide a comprehensive analysis of layout elements locating. In this paper, we first define three primary task categories: Long Document Understanding, numerical Reasoning, and cross-element Locating, and then propose a comprehensive benchmark, LongDocURL, integrating above three primary tasks and comprising 20 sub-tasks categorized based on different primary tasks and answer evidences. Furthermore, we develop a semi-automated construction pipeline and collect 2,325 high-quality question-answering pairs, covering more than 33,000 pages of documents, significantly outperforming existing benchmarks. Subsequently, we conduct comprehensive evaluation experiments on both open-source and closed-source models across 26 different configurations, revealing critical performance gaps in this field.

Figures (21)

• Figure 1: Comparison with other datasets in average pages and text tokens per document.
• Figure 2: LongDocURL comprises 20 sub-tasks focusing on three task categories: Understanding, numerical Reasoning, and cross-element Locating. (Top) Thumbnail of a document example. Orange boxes indicate answer evidence pages. (Bottom) Data examples generated from the document and screenshots of relevant part of answer evidence pages.
• Figure 3: Overview of our semi-automated construction pipeline. The pipeline comprises four modules: (a) Extract & Filter; (b) QA Generation; (c) Automated Verification; (d) Human Verification.
• Figure 4: Our LongDocURL comprises 20 sub-tasks. Inner: divided by the primary task categories (Understanding, Reasoning, and Locating). Middle: divided by the number of answer evidence pages (Single-Page, Multi-Page), and the number of types of evidence elements (Cross-Element). Outer: divided by the types of evidence elements (Text, Table, Figure, Layout).
• Figure 5: Fine-grained Results. We choose 3 proprietary and 3 open-source models to conduct further analysis based on (left) task types, document elements, evidence pages, and (right) document sources.