Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs

TL;DR

Grasp Any Region (GAR) addresses the gap in region-level understanding for multimodal LLMs by preserving global context while enabling precise, region-focused reasoning. It introduces RoI-aligned feature replay to couple a global feature map with region-specific details, and supports interactions among multiple prompts for compositional reasoning, framed within a single-pass vision encoder and an LLM. GAR-Bench provides structured evaluation of single-region comprehension and cross-prompt reasoning across multiple regions, including multi-task captioning and VQA. Experiments show state-of-the-art performance on DLC-Bench, GAR-Bench-VQA, and zero-shot transfer to VideoRefer benchmarks, indicating strong generalization to video and dense visual scenes.

Abstract

While Multimodal Large Language Models (MLLMs) excel at holistic understanding, they struggle in capturing the dense world with complex scenes, requiring fine-grained analysis of intricate details and object inter-relationships. Region-level MLLMs have been a promising step. However, previous attempts are generally optimized to understand given regions in isolation, neglecting crucial global contexts. To address this, we introduce Grasp Any Region (GAR) for comprehen- sive region-level visual understanding. Empowered by an effective RoI-aligned feature replay technique, GAR supports (1) precise perception by leveraging necessary global contexts, and (2) modeling interactions between multiple prompts. Together, it then naturally achieves (3) advanced compositional reasoning to answer specific free-form questions about any region, shifting the paradigm from passive description to active dialogue. Moreover, we construct GAR-Bench, which not only provides a more accurate evaluation of single-region comprehension, but also, more importantly, measures interactions and complex reasoning across multiple regions. Extensive experiments have demonstrated that GAR-1B not only maintains the state-of-the-art captioning capabilities, e.g., outperforming DAM-3B +4.5 on DLC-Bench, but also excels at modeling relationships between multiple prompts with advanced comprehension capabilities, even surpassing InternVL3-78B on GAR-Bench-VQA. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer-BenchQ, indicating its strong capabilities can be easily transferred to videos.

Figures (15)

• Figure 1: Performance comparison. GAR achieves strong performances not only on region-level understanding, but also on general multimodal benchmarks.
• Figure 2: Illustration of our GAR, which is superior at leveraging necessary global context to (a) generate precise captions, where green is correct and red means wrong, (b) model complex interactions among multiple prompts, and perform reasoning such as (c) recognizing non-entities. Colors of <Prompt0>, <Prompt1>, and <Prompt2> correspond to masks with respective colors. Images are sampled from shao2019objects365, lin2014microsoft, and mei2021depth for (a), (b), and (c), respectively.
• Figure 3: Illustration of our GAR. It leverages a single-pass visual encoder to create a holistic feature map of the entire scene, thus preserving global context. Simultaneously, an "RoI-Aligned Feature Replay" mechanism extracts high-fidelity features for specific objects of interest. Both the global context features and the detailed local features are then fed into an LLM to accurately infer complex relationships and interactions between multiple objects.
• Figure 4: Illustration of our training data pipeline, which mainly includes two rounds of captioning and judging. Specifically, (1) starting from using the seed dataset to train a seed captioner, we first construct 456K fine-grained descriptions. Subsequently, (2) we utilize both datasets to obtain a fine-grained captioner, and leverage the annotations of the Panoptic Scene Graph (PSG) dataset yang2022panoptic to provide sufficient relation-aware captions and question-answering pairs. Finally, our GAR models are trained with all three parts.
• Figure 5: Qualitative comparisons on DLC-Bench lian2025describe, where green indicates correct descriptions and red means errors. We compare our GAR-8B with DAM-3B lian2025describe. Thanks to the encoded global contexts, our GAR-8B produces much more accurate descriptions.