Model Composition for Multimodal Large Language Models

TL;DR

This paper tackles the resource-intensive challenge of building versatile Multimodal Large Language Models (MLLMs) by introducing a training-free model composition paradigm. The authors propose NaiveMC to reuse modality encoders and merge LLM parameters, and then enhance it with DAMC, which decouples modality-specific and language-model parameters and applies adaptive parameter adjustment to mitigate interference, formalized as $\theta_{\text{merge}} = \sum_{i=1}^{N} \lambda_i \theta_i$. To evaluate this approach, they introduce MCUB, a benchmark for cross-modality commonality understanding across image, audio, video, and point cloud modalities. Empirical results on MCUB and additional multimodal tasks show that DAMC achieves state-of-the-art performance, with significant gains as more modalities are incorporated and when compared to strong baselines like ImageBind-LLM and X-InstructBLIP. The work provides a scalable, data-efficient path to extend multimodal capabilities, with clear avenues for applying model composition to new modalities and larger models.

Abstract

Recent developments in Multimodal Large Language Models (MLLMs) have shown rapid progress, moving towards the goal of creating versatile MLLMs that understand inputs from various modalities. However, existing methods typically rely on joint training with paired multimodal instruction data, which is resource-intensive and challenging to extend to new modalities. In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters. Furthermore, we introduce DAMC to address parameter interference and mismatch issues during the merging process, thereby enhancing the model performance. To facilitate research in this area, we propose MCUB, a benchmark for assessing ability of MLLMs to understand inputs from diverse modalities. Experiments on this benchmark and four other multimodal understanding tasks show significant improvements over baselines, proving that model composition can create a versatile model capable of processing inputs from multiple modalities.

Figures (7)

• Figure 1: Illustration of various approaches for multimodal large language models: (a) aligning LLM with a multimodal encoder and (b) joint training with multiple modal encoders and (c) our proposed model composition method that creates a versatile model from existing MLLMs through a training-free and extensible process.
• Figure 2: Illustration of the model composition processes with only image and audio modalities are considered for simplicity. (a) and (b) show a basic model composition framework as described in Section \ref{['sec:naivemc']}, while (c) and (d) demonstrate model composition with parameter decoupling, as detailed in Section \ref{['sec:pd']}.
• Figure 3: An example of MCUB-4, where the objective is to identify common attributes from inputs across four different modalities.
• Figure 4: Qualitative examples on multimodal understanding of a composite model integrating four MLLMs.
• Figure 5: Additional qualitative results.