AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization

TL;DR

This work tackles the challenge of merging heterogeneous multimodal LLMs with differing architectures. AdaMMS introduces a mapping-based framework that aligns parameters across models, followed by adaptive linear interpolation to merge capabilities, and an unsupervised coefficient search guided by generation-consistency signals, all without labeled data. Empirical results on two heterogeneous MLLM pairs (Qwen2-based and LLaMA-based) show AdaMMS improves over strong baselines on vision-language benchmarks, with average gains and robustness to data subset size. The approach reduces data and compute requirements for model fusion in heterogeneous settings and offers a practical path to integrating diverse MLLMs in real-world applications.

Abstract

Recently, model merging methods have demonstrated powerful strengths in combining abilities on various tasks from multiple Large Language Models (LLMs). While previous model merging methods mainly focus on merging homogeneous models with identical architecture, they meet challenges when dealing with Multimodal Large Language Models (MLLMs) with inherent heterogeneous property, including differences in model architecture and the asymmetry in the parameter space. In this work, we propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs. Our method tackles the challenges in three steps: mapping, merging and searching. Specifically, we first design mapping function between models to apply model merging on MLLMs with different architecture. Then we apply linear interpolation on model weights to actively adapt the asymmetry in the heterogeneous MLLMs. Finally in the hyper-parameter searching step, we propose an unsupervised hyper-parameter selection method for model merging. As the first model merging method capable of merging heterogeneous MLLMs without labeled data, extensive experiments on various model combinations demonstrated that AdaMMS outperforms previous model merging methods on various vision-language benchmarks.

Figures (5)

• Figure 1: (a) Illustration of three steps in AdaMMS: Step-1, mapping MLLMs with different model architecture; Step-2, merging MLLMs with linear interpolation; Step-3, searching for optimal merging hyper-parameter by approximate task performance through generation consistency without labeled data. (b) The gain performance of AdaMMS on a broad range of multimodal tasks in comparison with existing merging approaches. Gain refers to the improvement obtained by subtracting the average result from the result of the fused model on a certain task. The result here is the average of the gains from the two MLLM pairs merging.
• Figure 2: Results on merging LLaVA-v1.5-7B into Qwen2-VL-7B. The $\alpha$ with the best perfo, bb=0 0 461 346rmance are the same as the $\alpha$ with the fewest response differences.
• Figure 3: Model responses with the change of $\alpha$ in linear interpolation. Similar colors indicate similar responses.
• Figure 4: Results on linear interpolation at different granularities of $\alpha$ when merging LLaVA-OneVison-7B into Qwen2-VL-7B-7B. (Left: MME, Right: OCRBench)
• Figure 5: Generation consistency and model performance (score) for MME, MMMU, OCRBench and SeedBench when merging LLaVA-OneVision-7B into Qwen2-VL-7B. Generation consistency is calculated as the reciprocal of the sum of different responses from models with adjacent $\alpha$ candidates. The horizontal axis is the $\alpha$ of the linear interpolation.