VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning

TL;DR

VRAG-RL tackles the challenge of reasoning over visually rich information by integrating a visual perception action space with a retrieval-informed reward and iterative GRPO-based reinforcement learning. The framework enables coarse-to-fine sampling of information, scalable trajectory data via multi-expert sampling, and a three-component reward that aligns retrieval and reasoning with final answers. Empirical results across three benchmarks show substantial improvements over text- and vision-based baselines, with ablations validating the contributions of the perception actions and reward design. This work offers a practical pathway to robust, multi-turn visually grounded RAG, with potential impact on document understanding and other visually dense information tasks.

Abstract

Effectively retrieving, reasoning and understanding visually rich information remains a challenge for RAG methods. Traditional text-based methods cannot handle visual-related information. On the other hand, current vision-based RAG approaches are often limited by fixed pipelines and frequently struggle to reason effectively due to the insufficient activation of the fundamental capabilities of models. As RL has been proven to be beneficial for model reasoning, we introduce VRAG-RL, a novel RL framework tailored for complex reasoning across visually rich information. With this framework, VLMs interact with search engines, autonomously sampling single-turn or multi-turn reasoning trajectories with the help of visual perception tokens and undergoing continual optimization based on these samples. Our approach highlights key limitations of RL in RAG domains: (i) Prior Multi-modal RAG approaches tend to merely incorporate images into the context, leading to insufficient reasoning token allocation and neglecting visual-specific perception; and (ii) When models interact with search engines, their queries often fail to retrieve relevant information due to the inability to articulate requirements, thereby leading to suboptimal performance. To address these challenges, we define an action space tailored for visually rich inputs, with actions including cropping and scaling, allowing the model to gather information from a coarse-to-fine perspective. Furthermore, to bridge the gap between users' original inquiries and the retriever, we employ a simple yet effective reward that integrates query rewriting and retrieval performance with a model-based reward. Our VRAG-RL optimizes VLMs for RAG tasks using specially designed RL strategies, aligning the model with real-world applications. The code is available at https://github.com/Alibaba-NLP/VRAG.

Figures (12)

• Figure 1: Overall Framework of our Reinforcement Learning Framework. (a) demonstrates the interaction process between the model and the external environment, as well as the implementation of the GRPO algorithm. (b) shows the proposed visual perception action space which allows the model to extract information from a coarse-to-fine perspective. (c) is the specially designed reward for RAG, which combines outcome and retrieval performance across the entire sampling process.
• Figure 2: Comparison between our VRAG-RL and the traditional RAG in terms of perception methods. (a) Traditional methods lack effective perception, which easily leads to repetitive and ineffective retrieval calls and suboptimal outcomes. (b) Our VRAG-RL is efficient and accurate, enabling the model to perceive information-dense regions from a coarse-to-fine perspective.
• Figure 3: Experiments on the impact of context length on model performance.
• Figure 4: Retrieval performance of our approach.
• Figure 5: Relative performance on MMLongBench.