RegionReasoner: Region-Grounded Multi-Round Visual Reasoning

TL;DR

RegionReasoner tackles multi-round, region-grounded visual reasoning by introducing RegionDial-Bench and a reinforcement learning framework that enforces explicit region citations and global–local semantic alignment across turns. The model outputs four-block trajectories per turn—<scene>, <focus>, <think>, <answer>—and is trained with a GRPO objective combining a reference-citation reward and a global–local consistency reward, optimizing over $T$ turns with per-turn reward $R(t)$ and episode return $\sum_t R(t)$. On RegionDial-Bench, RegionReasoner-7B outperforms strong baselines for both referring detection and segmentation, with especially strong gains as dialogue depth increases, demonstrating reduced drift and improved grounding fidelity. The work provides a scalable, interpretable baseline for reference-grounded multi-round reasoning and introduces a challenging benchmark that couples detection and segmentation under iterative, dialogue-style supervision.

Abstract

Large vision-language models have achieved remarkable progress in visual reasoning, yet most existing systems rely on single-step or text-only reasoning, limiting their ability to iteratively refine understanding across multiple visual contexts. To address this limitation, we introduce a new multi-round visual reasoning benchmark with training and test sets spanning both detection and segmentation tasks, enabling systematic evaluation under iterative reasoning scenarios. We further propose RegionReasoner, a reinforcement learning framework that enforces grounded reasoning by requiring each reasoning trace to explicitly cite the corresponding reference bounding boxes, while maintaining semantic coherence via a global-local consistency reward. This reward extracts key objects and nouns from both global scene captions and region-level captions, aligning them with the reasoning trace to ensure consistency across reasoning steps. RegionReasoner is optimized with structured rewards combining grounding fidelity and global-local semantic alignment. Experiments on detection and segmentation tasks show that RegionReasoner-7B, together with our newly introduced benchmark RegionDial-Bench, considerably improves multi-round reasoning accuracy, spatial grounding precision, and global-local consistency, establishing a strong baseline for this emerging research direction.

Figures (7)

• Figure 1: RegionReasoner in a three–round, region-grounded dialogue. At round $t$, the user query may refer to a region localized earlier (R1/R2). For each turn, RegionReasoner produces a structured trajectory: <scene> (global context), <focus> (caption restricted to the referenced region with serialized coordinates, e.g., bbox=[x$_1$,y$_1$,x$_2$,y$_2$]), <think> (reasoning that explicitly cites the reference and the required spatial relation), and <answer> (final localization). The example shows correct citation and stable multi-round grounding for “behind the R1 on the left” and “next to the R2”, illustrating how explicit reference use and coherent global–local descriptions support consistent localization as the dialogue deepens.
• Figure 2: Qualitative multi-round trajectories (R1–R3) on our RegionDial-Bench . Each panel shows RegionReasoner vs. VisionReasoner. Blue boxes mark the referenced region passed from the previous round; yellow boxes denote the predicted target at the current round; the right column lists ground-truth labels. RegionReasoner consistently cites the reference coordinates inside <think> and aligns its reasoning with global (<scene>) and local (<focus>) descriptions, yielding stable localization in later rounds. VisionReasoner, lacking explicit citation, is prone to semantic drift or neighbor confusion when context accumulates.
• Figure 3: Example of training data construction. Round 1 localizes the "laying dog" (red box). Round 2 reformulates the query into "standing dog, next to bbox=[0,457,374,672]" (blue box).
• Figure 4: Example from RefCOCO+ Multi-turn illustrating the construction pipeline in RegionDial-Bench. Round 1 localizes the "man in blue shirt" (red box) with ground-truth box [47,107,303,466]. This box is then injected into Round 2 as an explicit reference, reformulating the query into "Who is next to bbox=[47,107,303,466]?" to localize the "man in white shirt" (blue box).
• Figure 5: Framework of RegionReasoner. The model processes multi-round queries with Qwen2.5-VL-7B, guided by two complementary reward signals: (1) the reference citation reward, ensuring explicit grounding to previously predicted objects, and (2) the global–local consistency reward, enforcing alignment between holistic and reference-based reasoning.