RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension

TL;DR

RegionBLIP presents a unified, incremental pre-training framework that extends a frozen Q-Former-based BLIP-2 backbone to regional object understanding and point-cloud modalities via modality-specific LoRA adapters and a novel PaFE module that aligns region features with text. A one-stage pre-training objective combines multimodal alignment with LLM reasoning, and an incremental strategy avoids costly re-training on large image-text corpora while adding image-region-text, point-cloud-text, and point-cloud-region-text data. The approach preserves image comprehension capabilities and demonstrates strong regional and 3D object captioning performance, aided by the RegionCap-10M dataset built from large-scale image collections. RegionBLIP thus enables efficient, scalable expansion of MLLMs to holistic and regional modalities, with practical impact for multimodal understanding in vision-language systems, robotics, and AR/VR applications. $\mathcal{L} = \mathcal{L}_{ITC} + \mathcal{L}_{ITG} + \mathcal{L}_{ITM} + \mathcal{L}_{LLM} + \lambda \mathcal{L}_{reg}$, where $\lambda=1.0$, captures the core training objective.$

Abstract

In this work, we investigate extending the comprehension of Multi-modal Large Language Models (MLLMs) to regional objects. To this end, we propose to extract features corresponding to regional objects as soft prompts for LLM, which provides a straightforward and scalable approach and eliminates the need for LLM fine-tuning. To effectively extract regional features from regular image features and irregular point cloud features, we present a novel and unified position-assisted feature extraction module. Furthermore, training an MLLM from scratch is highly time-consuming. Thus, we propose incrementally extending existing pre-trained MLLMs to comprehend more modalities and the regional objects of those modalities. Specifically, we freeze the Q-Former from BLIP-2, an impressive MLLM, and optimize the modality-specific Lora parameters in Q-Former and LLM for each newly introduced modality. The freezing of the Q-Former eliminates the need for extensive pre-training on massive image-text data. The freezed Q-Former pre-trained from massive image-text data is also beneficial for the pre-training on image-region-text data. We name our framework RegionBLIP. We pre-train RegionBLIP on image-region-text, point-cloud-text, and point-cloud-region-text data. Experimental results verify that \Ours{} can preserve the image comprehension capability of BILP-2 and further gain a comprehension of the newly introduced point cloud modality and regional objects. The Data, Code, and Pre-trained models will be available at https://github.com/mightyzau/RegionBLIP.

Figures (7)

• Figure 1: RegionBLIP is a unified incremental pre-training framework supporting LLM's comprehension of images, point clouds, and regional objects. For efficient pre-training, RegionBLIP freezes the Q-Former of BLIP-2 blip2_2023 and learns a set of modality-specific Lora parameters for newly added modalities. To effectively extract region features from regular image features and irregular point cloud features, RegionBLIP proposes a unified scheme of position-assisted region feature extraction module.
• Figure 2: Examples of image captioning. The samples are from the COCO caption coco_LinMBHPRDZ14 test set, and the model is RegionBLIP OPT$_{2.7B}$.
• Figure 3: Examples of image-region captioning. The samples are from the RefCOCO refcoco test set, and the model is RegionBLIP OPT$_{2.7B}$.
• Figure 4: Examples of point cloud captioning. The samples are from the Objaverse objaverse_abs-2212-08051 test set, and the model is RegionBLIP OPT$_{2.7B}$.
• Figure 5: Examples of point-cloud-region captioning. The samples are from the ScanRefer scanrefer_ChenCN20 validation set, and the model is RegionBLIP OPT$_{2.7B}$. In this work, we did not utilize the color information of the point cloud, which limits the performance of point cloud region captioning to some extent.