PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs

TL;DR

This work proposes a plateau-guided model merging method that selectively injects base language model parameters into MLLMs and reveals a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation.

Abstract

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.

Figures (9)

• Figure 1: Our proposed PlaM improves visual grounding and prediction. Given the clock image (left) and the question "It is (_) past six.", the original MLLM without PlaM attends diffusely and answers “half” (middle). After applying PlaM model merging, attention concentrates on the clock hands and the model correctly answers “quarter” (right).
• Figure 2: Performance versus cut layer ($k$) under depth-controlled vision token masking. Vision tokens are removed from layer $k$ onward ($k=L+1$ indicates no masking), and each curve reports the official metric on each benchmark. See \ref{['tab:llavav1.5', 'tab:Qwen2.53B', 'tab:Qwen2.57B', 'tab:Qwen34B', 'tab:Qwen38B']} for the corresponding numerical results (Appendix \ref{['app-exp']}).
• Figure 3: Overall comparison of performance and attention mass before and after merging. Left: Layer-wise vision token masking results, where the model performance is measured by progressively removing vision tokens from layer $k$ to the last layer. The red marker indicates the selected merge start layer $k_0$, beyond which visual information yields diminishing performance gains and merging is applied. Right: Layer-wise attention mass profiles comparing the original model (Base) and the merged model (Merged). The shaded region denotes the merged layers $\{k_0,L\}$. Results are shown for LLaVA-v1.5-7B on SEED-Bench-2-Plus (top row) and Qwen2.5-VL-3B-Instruct on RealWorldQA (bottom row).
• Figure 4: Attention heatmaps for LLaVA-v1.5-7B (bottom) and its PlaM-merged model (top). Additional case studies for other backbones are provided in Appendix \ref{['figure:heapmap_llava', 'figure:heapmap_qwen253', 'figure:heapmap_qwen257', 'figure:heapmap_qwen34', 'figure:heapmap_qwen38']}.
• Figure 5: Attention heatmaps for LLaVA-v1.5-7B (bottom) and its PlaM-merged model (top).