GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

TL;DR

GDI-Bench addresses the need for robust evaluation of multimodal document understanding by decoupling visual complexity and reasoning complexity into V0-V2 and R0-R2, respectively, and by grading difficulty across 19 tasks in 9 scenarios. The authors introduce Layer-wise Adaptive Freeze-Tuning (LW-AFT) to mitigate catastrophic forgetting during supervised fine-tuning and propose a GDI-Model that achieves state-of-the-art performance on GDI-Bench and other benchmarks. The benchmark combines 2.3k images with a two-layer annotation pipeline (OmniDocBench and in-house data) and a mixed-data generation strategy (GPT-4o plus rule-based tasks) to yield 3,660 test cases with rigorous quality control. The work demonstrates meaningful weaknesses in current models at higher reasoning levels and shows that LW-AFT can preserve generalization while enabling domain-specific improvements, with code and models released on HuggingFace for open community use.

Abstract

The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 2.3k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate various open-source and closed-source models on GDI-Bench, conducting decoupled analyses in the visual and reasoning domains, revealing their strengths and weaknesses. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI-Model that mitigates catastrophic forgetting during the supervised fine-tuning (SFT) process through an intelligence-preserving training strategy, thereby reinforcing the inherent weaknesses of the base model. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and models are or will be open-sourced on https://huggingface.co/GDIBench.

Figures (17)

• Figure 1: Overview of GDI-Bench. The benchmark decouples document understanding complexity into visual complexity (V0-V2) and reasoning complexity (R0-R2) dimensions, creating a comprehensive evaluation framework for assessing MLLMs' capabilities across various document types and reasoning tasks. Queries marked with a “CN” tag originate in Chinese and have been translated into English using Google Translate.
• Figure 1: Complexity Distribution in the GDI Benchmark.
• Figure 2: Performance of various open-source and closed-source models on GDI-Bench at different levels of reasoning complexity. The GDI-Model is fine-tuned based on the InternVL3-8B model.
• Figure 3: Distribution of visual complexity scores across nine document categories as reported in the OmniDocBench. The SOTA performance results from the benchmark show a substantial gap between document types, providing empirical justification for our visual complexity taxonomy.
• Figure 4: Annotation Process of GDI-Bench.