Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models

TL;DR

<3-5 sentence high-level summary> Griffon-G tackles the challenge of unifying vision-language and vision-centric tasks within a single large multimodal model by introducing CCMD-8M, a multi-dimension curated dataset, and a Paradigm Progressive Learning Pipeline to stabilize joint optimization. The approach yields state-of-the-art or competitive results across VQA, scene-text, document-VQA, REC, and object detection benchmarks, demonstrating effective high-resolution perception alongside precise localization. Key contributions include dataset construction with task- and annotation-level curation, a three-stage training regimen, and extensive ablations validating the mutual benefits between vision-language and vision-centric capabilities. This work advances toward a general, instruction-following, multimodal foundation model with broad practical impact in complex real-world tasks.

Abstract

Large Multimodal Models (LMMs) have achieved significant breakthroughs in various vision-language and vision-centric tasks based on auto-regressive modeling. However, these models typically focus on either vision-centric tasks, such as visual grounding and region description, or vision-language tasks, like image caption and multi-scenario VQAs. None of the LMMs have yet comprehensively unified both types of tasks within a single model, as seen in Large Language Models in the natural language processing field. Furthermore, even with abundant multi-task instruction-following data, directly stacking these data for universal capabilities extension remains challenging. To address these issues, we introduce a novel multi-dimension curated and consolidated multimodal dataset, named CCMD-8M, which overcomes the data barriers of unifying vision-centric and vision-language tasks through multi-level data curation and multi-task consolidation. More importantly, we present Griffon-G, a general large multimodal model that addresses both vision-centric and vision-language tasks within a single end-to-end paradigm. Griffon-G resolves the training collapse issue encountered during the joint optimization of these tasks, achieving better training efficiency. Evaluations across multimodal benchmarks, general Visual Question Answering (VQA) tasks, scene text-centric VQA tasks, document-related VQA tasks, Referring Expression Comprehension, and object detection demonstrate that Griffon-G surpasses the advanced LMMs and achieves expert-level performance in complicated vision-centric tasks.

Figures (5)

• Figure 1: Overview of the Griffon-G's key capabilities. Griffon-G extends the ability boundary of general LMMs to excel in both vision-centric and vision-language tasks with one unified framework.
• Figure 2: Structure comparison among vision-language LMMs, vision-centric LMMs, and Griffon-G. Griffon-G achieves high-resolution understanding and fine-grained localization with a streamlined high-resolution architecture without complex slicing operations and additional structures.
• Figure 3: Illustration of the curation process of CCMD-8M Dataset. We conduct task-level curation for REC data and annotation-level curation for REG data to improve the information density, diversity, and quality.
• Figure 4: Visualization of Paradigm Progressive Learning Pipeline. With our proposed PPLP, the loss of the instruction tuning stage converges as normal without the training collapse when generalizing beyond vision-language tasks to vision-centric tasks.
• Figure 5: Qualitative analysis with more test samples on general scenarios and scenarios with dense text. Griffon-G has demonstrated impressive performance when compared to GPT-4o. The red color indicates the wrong responses.