Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models

TL;DR

This work addresses the challenge that large multimodal language models underperform on language-only tasks when trained under developmentally plausible, data-limited conditions. It introduces a training-free inference-time model merging approach that linearly combines parameters from a language-only model and a multimodal model, aimed at preserving linguistic competencies without sacrificing multimodal capabilities. Empirical results show strong language-only performance on BLiMP and competitive multimodal results, with merging offering a practical means to balance both domains under BabyLM constraints. The findings highlight the potential of parameter-space fusion to enable lighter, cognitively plausible models that maintain cross-modal functionality, while outlining computational and data limitations to inspire future work.

Abstract

State-of-the-art vision-and-language models consist of many parameters and learn from enormous datasets, surpassing the amounts of linguistic data that children are exposed to as they acquire a language. This paper presents our approach to the multimodal track of the BabyLM challenge addressing this discrepancy. We develop language-only and multimodal models in low-resource settings using developmentally plausible datasets, with our multimodal models outperforming previous BabyLM baselines. One finding in the multimodal language model literature is that these models tend to underperform in \textit{language-only} tasks. Therefore, we focus on maintaining language-only abilities in multimodal models. To this end, we experiment with \textit{model merging}, where we fuse the parameters of multimodal models with those of language-only models using weighted linear interpolation. Our results corroborate the findings that multimodal models underperform in language-only benchmarks that focus on grammar, and model merging with text-only models can help alleviate this problem to some extent, while maintaining multimodal performance.

Figures (11)

• Figure 1: Weighted merging of language-only and multimodal models in the form of linear interpolation.
• Figure 2: An example from Winoground. Left: 'painting the white wall red'. Right: 'painting the red wall white'.
• Figure 3: Average accuracies for the text-only model and the multimodal model over the training checkpoints, for the BLiMP full benchmark.
• Figure 4: Average accuracies for the text-only model and the multimodal model over the training checkpoints, for the BLiMP fast benchmark.
• Figure 5: Average accuracies for the text-only model and the multimodal model over the training checkpoints, for the BLiMP supplement fast benchmark.