Visual Instruction Tuning with Chain of Region-of-Interest

TL;DR

CoRoI tackles the computational burden of high-resolution inputs in multimodal large language models by learning a Chain of Region-of-Interest that selects informative high-resolution patches guided by low-resolution visual tokens and a question. It then injects these ROI clips into the LLM hidden layers via a lightweight residual cross-attention mechanism, enabling tight HR-LR interactions without encoding all HR tokens. The method is trained in two stages with a frozen vision encoder and evaluated on 11 benchmarks across 7B–34B parameter models, consistently outperforming open-source baselines and achieving competitive results with proprietary models on several tasks. The results demonstrate strong cross-modal reasoning with reduced computation, suggesting practical benefits for scalable high-resolution vision-language systems. Limitations include reliance on upsampling-based HR generation and the current ceiling on high-resolution sizes, leaving room for further data and compute-driven improvements.

Abstract

High-resolution (HR) images are pivotal for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs). However, directly increasing image resolution can significantly escalate computational demands. In this study, we propose a method called Chain of Region-of-Interest (CoRoI) for Visual Instruction Tuning, aimed at alleviating the computational burden associated with high-resolution images for MLLMs. Drawing inspiration from the selective nature of the human visual system, we recognize that not all regions within high-resolution images carry equal importance. CoRoI seeks to identify and prioritize the most informative regions, thereby enhancing multimodal visual comprehension and recognition while circumventing the need for processing lengthy HR image tokens. Through extensive experiments on 11 benchmarks, we validate the efficacy of CoRoI across varying sizes, ranging from 7B to 34B in parameters. Our models consistently demonstrate superior performance across diverse multimodal benchmarks and tasks. Notably, our method outperforms LLaVA-NeXT on almost all benchmarks and our finetuned 34B model surpasses proprietary methods like Gemini Pro 1.0 on six benchmarks, as well as outperforming GPT-4V on MMB, SEED-I, and MME.

Figures (5)

• Figure 1: Zero-shot performance comparison with LLaVA-1.5, LLaVA-NeXT, Gemini-Pro 1.0, and GPT-4V.
• Figure 2: Illustration of CoRoI. Low-resolution images and text clues are used to identify a chain of regions of interest in the corresponding high-resolution images. These regions are then injected into the hidden states of the LLMs to facilitate answer generation. Here, $m=1$, $\lambda=2$, $n=3$.
• Figure 3: CoRoI enables the model to extract a chain of region-of-interest during inference. We used variant: CoRoI-v2 (Hermes-2-Yi-34B). The selected regions are closely relevant to the asked questions. We also provide the answers given by GPT-4V.
• Figure 4: Modules design details.
• Figure 5: Reasoning Generative Capability. The generated images are from DALLE3 using prompts generated by CoRoI.