Unified Generative and Discriminative Training for Multi-modal Large Language Models

TL;DR

Vision-language models trained separately for generation or discrimination struggle with complex interleaved inputs and fine-grained semantics. Sugar introduces a structure-induced joint training framework that enforces semantic relationships across interleaved image-text sequences using a Dynamic Time Warping–based global alignment kernel and a Triple Kernel for fine-grained differentiation, yielding a balanced generative-discriminative model. Empirically, Sugar achieves state-of-the-art results on challenging generative tasks and excels in interleaved and fine-grained retrieval, while enabling retrieval-augmented generation within a single model. This approach offers a unified path to more robust, capable multimodal reasoning and knowledge integration.

Abstract

In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM's hidden state. This approach enhances the MLLM's ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

Figures (14)

• Figure 1: (a) In WebQA WebQA21, the accuracy roughly forms a “U” shape curve when the relevant image-text pair for a question appears at different positions. While our model also shows similar trends, it tends to be more stable overall. (b) The accuracy of various types of questions in MMVP-VLM tong2024eyes, it can be observed that our model's performance improves on such tasks after introducing the discriminative training. Details can be seen in Appendix \ref{['app_intro']}
• Figure 2: Our structure-induced generative and discriminative training joint training strategy.
• Figure 3: (a) Dynamic Sequence Alignment. Semantically matched slices are connected with a blue dashed line. The arrows indicate the direction of the ordered temporal alignment path. With these alignments, we can obtain the similarity between two interleaved inputs for training. (b) Sugar Framework. Sugar supports both multi-modal generation and retrieval simultaneously.
• Figure 4: Selected examples for various image-text tasks. The HTML]FCF3F3pink background indicates retrieval results, while the HTML]D4E5F7blue background indicates generated results. More examples are provided in the Appendix \ref{['app_quality']}.
• Figure 5: A Case for WebQA. The index for the useful pair is three.

Theorems & Definitions (6)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof