Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

TL;DR

The paper addresses whether multimodal instruction tuning degrades language reasoning and finds degradation is strongly influenced by the chosen base LLM. It analyzes LLaVA variants across eight language and five vision tasks, showing that stronger LLMs may degrade more, while some tasks (e.g., CommonsenseQA) improve. To mitigate degradation without retraining, it introduces a training-free model merging approach that forms a merged model using a task vector $T = \theta_{vlm} - \theta_{llm}$ and merged weights $M = \alpha \hat{T} + \theta_{vlm}$, with performance depending on the merge weight $\alpha$ and sparsity $K$. The results demonstrate substantial recovery of language reasoning and even gains in visual tasks at appropriate merging settings, offering a practical, deployment-friendly solution. This work highlights the potential of merging-based mitigation to balance language and visual reasoning in multimodal systems without additional training.

Abstract

Multimodal models typically combine a powerful large language model (LLM) with a vision encoder and are then trained on multimodal data via instruction tuning. While this process adapts LLMs to multimodal settings, it remains unclear whether this adaptation compromises their original language reasoning capabilities. In this work, we explore the effects of multimodal instruction tuning on language reasoning performance. We focus on LLaVA, a leading multimodal framework that integrates LLMs such as Vicuna or Mistral with the CLIP vision encoder. We compare the performance of the original LLMs with their multimodal-adapted counterparts across eight language reasoning tasks. Our experiments yield several key insights. First, the impact of multimodal learning varies between Vicuna and Mistral: we observe a degradation in language reasoning for Mistral but improvements for Vicuna across most tasks. Second, while multimodal instruction learning consistently degrades performance on mathematical reasoning tasks (e.g., GSM8K), it enhances performance on commonsense reasoning tasks (e.g., CommonsenseQA). Finally, we demonstrate that a training-free model merging technique can effectively mitigate the language reasoning degradation observed in multimodal-adapted Mistral and even improve performance on visual tasks.

Figures (1)

• Figure 1: (a) Comparison of the performance of Mistral and Vicuna LLMs on eight language reasoning tasks with their corresponding LVLMs. (b) Average percentage of performance changes across 8 language tasks and 5 vision tasks using different merging weights.