GroundingGPT:Language Enhanced Multi-modal Grounding Model

TL;DR

<3-5 sentence high-level summary> GroundingGPT addresses the limitation of existing multi-modal large language models in capturing fine-grained cross-modal information. It introduces modality-specific adapters that map image, video, and audio features into a unified LLM embedding space and employs a three-stage coarse-to-fine training pipeline with stage-specific data to enable precise spatial-temporal grounding. The approach demonstrates strong performance on image and video grounding tasks, multi-modal understanding benchmarks, and a reduction in object hallucination, validated across REC, temporal grounding, VQA-style tasks, and POPE. By providing end-to-end grounding across modalities and releasing code and data, it offers a practical, scalable path toward more accurate and reliable multi-modal AI systems.

Abstract

Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.

Figures (11)

• Figure 1: The overall structure of GroundingGPT involves separate encoders and adapters for each modality. Blue boxes represent video inputs, yellow boxes represent image inputs, and pink boxes represent audio inputs.
• Figure 2: The data construction pipeline and examples for the last two training stages. To simplify, the multi-turn conversation examples only showcase two rounds of question-answer interactions.
• Figure 3: Qualitative results of GroundingGPT on multi-modal grounding tasks.
• Figure 4: The system message and in-context example used for generating detailed description dataset.
• Figure 5: The system message and in-context example used for generating conversation dataset.