DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understanding

TL;DR

DocR1 tackles multi-page document understanding by marrying a specialized RL framework, EviGRPO, with an evidence-grounded coarse-to-fine reasoning strategy. It introduces a two-stage data annotation pipeline and two datasets (EviBench and ArxivFullQA) to enable high-quality supervision at limited scale, and uses curriculum training to transfer single-page reasoning to multi-page tasks. Empirical results show state-of-the-art performance on multi-page benchmarks and competitive performance on single-page tasks, with strong evidence grounding and interpretability. The work demonstrates the value of RL for document-level reasoning in multimodal models and provides datasets and training paradigms to promote data-efficient learning in complex, real-world scenarios.

Abstract

Understanding multi-page documents poses a significant challenge for multimodal large language models (MLLMs), as it requires fine-grained visual comprehension and multi-hop reasoning across pages. While prior work has explored reinforcement learning (RL) for enhancing advanced reasoning in MLLMs, its application to multi-page document understanding remains underexplored. In this paper, we introduce DocR1, an MLLM trained with a novel RL framework, Evidence Page-Guided GRPO (EviGRPO). EviGRPO incorporates an evidence-aware reward mechanism that promotes a coarse-to-fine reasoning strategy, guiding the model to first retrieve relevant pages before generating answers. This training paradigm enables us to build high-quality models with limited supervision. To support this, we design a two-stage annotation pipeline and a curriculum learning strategy, based on which we construct two datasets: EviBench, a high-quality training set with 4.8k examples, and ArxivFullQA, an evaluation benchmark with 8.6k QA pairs based on scientific papers. Extensive experiments across a wide range of benchmarks demonstrate that DocR1 achieves state-of-the-art performance on multi-page tasks, while consistently maintaining strong results on single-page benchmarks.

Figures (7)

• Figure 1: Our DocR1 has significant improvements on various benchmarks of multi-page document understanding.
• Figure 2: Our proposed EviGRPO training framework adopts a two-stage strategy to progressively enhance the model's multi-page reasoning capabilities.
• Figure 3: The two-stage data annotation pipeline consists of a data generation process and an annotation verification process, designed to ensure the quality of the annotations.
• Figure 4: Examples on multi-page document QA task. It can be seen that DocR1 can not only answer the questions correctly, but also provide relatively accurate evidence pages.
• Figure 5: Numpy-style pseudocode of the EviGRPO training loop used for DocR1.