Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling

TL;DR

This work introduces MMER, a training-free framework for expanding LLMs to handle additional modalities by merging LLM parameters across multiple MLLMs and then decoupling modality-specific components with binary masks. By computing a merged task vector $ au_*$ and modality-specific masks $m_i$, MMER reconstructs near-original modality capabilities while maintaining the base model's fidelity, significantly reducing parameter interference. The approach supports multi-modality expansion, retention of original performance (approximately 99%), and mitigation of catastrophic forgetting when adapted to new tasks. Empirical results across vision, audio, video, and point-cloud modalities demonstrate robust expansion, strong retention, and resilience to forgetting, with storage considerations balanced against performance gains. The method leverages a training-free paradigm that bridges the gap between pure model merging and full retraining, enabling scalable deployment of richer multimodal LLMs.

Abstract

Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs (MLLMs). However, this paradigm heavily relies on resource-intensive and inflexible fine-tuning from scratch with new multimodal data. In this paper, we propose MMER (Multi-modality Expansion and Retention), a training-free approach that integrates existing MLLMs for effective multimodal expansion while retaining their original performance. Specifically, MMER reuses MLLMs' multimodal encoders while merging their LLM parameters. By comparing original and merged LLM parameters, MMER generates binary masks to approximately separate LLM parameters for each modality. These decoupled parameters can independently process modality-specific inputs, reducing parameter conflicts and preserving original MLLMs' fidelity. MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks. Extensive experiments show significant improvements over baselines, proving that MMER effectively expands LLMs' multimodal capabilities while retaining 99% of the original performance, and also markedly mitigates catastrophic forgetting.

Figures (9)

• Figure 1: The key ideas of MMER. Multi-Modality Expansion creates a versatile model from existing MLLMs via a training-free, extensible process. Multi-Modality Retention reconstructs original or new task MLLMs to retain performance and mitigate catastrophic forgetting.
• Figure 2: The overview of MMER, considering only the Vision and Point Cloud modalities for clarity. Each block corresponds to the same weight matrix, with empty blocks denoting zero value. "$\approx$" signifies similar performance.
• Figure 3: Details of MMER's dynamic processing. and represent the Hadamard product and addition.
• Figure 4: Performance retention vs. MLLMs quantity.
• Figure 5: Parameters overlap across modalities.