GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models

TL;DR

GeM-VG addresses the lack of a unified model for generalized multi-image visual grounding by introducing MG-Data-240K and a hybrid RL finetuning strategy that blends chain-of-thought and direct answers. Built on a Qwen2-VL-based architecture, it outputs a set of bounding boxes across multiple images with their image indices, enabling arbitrary numbers of targets. Evaluations on MIG-Bench, MC-Bench, and ODINW demonstrate consistent gains in multi-image grounding, single-image grounding, and multi-image understanding, and ablations validate the effectiveness of the reward design and training strategy. The work provides a scalable path toward robust, generalized grounding for real-world multi-image tasks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive progress in single-image grounding and general multi-image understanding. Recently, some methods begin to address multi-image grounding. However, they are constrained by single-target localization and limited types of practical tasks, due to the lack of unified modeling for generalized grounding tasks. Therefore, we propose GeM-VG, an MLLM capable of Generalized Multi-image Visual Grounding. To support this, we systematically categorize and organize existing multi-image grounding tasks according to their reliance of cross-image cues and reasoning, and introduce the MG-Data-240K dataset, addressing the limitations of existing datasets regarding target quantity and image relation. To tackle the challenges of robustly handling diverse multi-image grounding tasks, we further propose a hybrid reinforcement finetuning strategy that integrates chain-of-thought (CoT) reasoning and direct answering, considering their complementary strengths. This strategy adopts an R1-like algorithm guided by a carefully designed rule-based reward, effectively enhancing the model's overall perception and reasoning capabilities. Extensive experiments demonstrate the superior generalized grounding capabilities of our model. For multi-image grounding, it outperforms the previous leading MLLMs by 2.0% and 9.7% on MIG-Bench and MC-Bench, respectively. In single-image grounding, it achieves a 9.1% improvement over the base model on ODINW. Furthermore, our model retains strong capabilities in general multi-image understanding.

Figures (6)

• Figure 1: (Top) The limitations of existing multi-image grounding models: (a) The reasoning-guided model struggles with fine-grained perception tasks, where the visual cues are hard to verbalize. (b) They fail to localize all referring instances in multi-object scenarios. (Bottom) Our model, GeM-VG, achieves strong performance across a range of multi-image grounding tasks.
• Figure 2: (Left) Overview of GeM-VG. GeM-VG is capable of reasoning over multiple visual contexts and localizing all referring instances. (Right) We provide a systematic taxonomy of generalized multi-image grounding scenarios.
• Figure 3: Framework of our reinforcement finetuning. Considering several failure types in multi-image grounding, we design a multi-dimensional rule-based reward. Before computing the advantages, we apply a reward adjustment strategy to facilitate joint optimization between both answering paradigms.
• Figure 4: Examples of training data within each type of tasks. For reasoning grounding task, we use the free-form grounding data from MGrounding-630K.
• Figure 5: Some visualization examples on grounding task. GeM-VG is capable of a wide range of grounding tasks.