Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?

Yijie Hu, Zihao Zhou, Kaizhu Huang, Xiaowei Huang, Qiufeng Wang

TL;DR

The paper tackles the challenge of enhancing multimodal math reasoning in MLLMs by directly absorbing capabilities from off-the-shelf Math LLMs without tuning. It identifies two key hurdles—finding math-reasoning–related parameters and aligning the parameter spaces of LLMs and MLLMs—and introduces IP-Merging, a tune-free method that identifies crucial layers via subspace similarity, rescales them, and projects them into the MLLM subspace before merging. Empirical results across MathVista, MathVerse, DynaMath, MathVision, and general QA benchmarks show consistent math reasoning gains with minimal disruption to non-math abilities, and ablations confirm the complementary benefits of selection, rescaling, and projection. The approach scales to larger and multiple models and demonstrates transferability across System-1 and System-2 LLMs, suggesting practical utility for augmenting multimodal reasoning without expensive data collection or retraining.

Abstract

Math reasoning has been one crucial ability of large language models (LLMs), where significant advancements have been achieved in recent years. However, most efforts focus on LLMs by curating high-quality annotation data and intricate training (or inference) paradigms, while the math reasoning performance of multi-modal LLMs (MLLMs) remains lagging behind. Since the MLLM typically consists of an LLM and a vision block, we wonder: Can MLLMs directly absorb math reasoning abilities from off-the-shelf math LLMs without tuning? Recent model-merging approaches may offer insights into this question. However, they overlook the alignment between the MLLM and LLM, where we find that there is a large gap between their parameter spaces, resulting in lower performance. Our empirical evidence reveals two key factors behind this issue: the identification of crucial reasoning-associated layers in the model and the mitigation of the gaps in parameter space. Based on the empirical insights, we propose IP-Merging that first identifies the reasoning-associated parameters in both MLLM and Math LLM, then projects them into the subspace of MLLM, aiming to maintain the alignment, and finally merges parameters in this subspace. IP-Merging is a tuning-free approach since parameters are directly adjusted. Extensive experiments demonstrate that our IP-Merging method can enhance the math reasoning ability of MLLMs directly from Math LLMs without compromising their other capabilities.

Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?

TL;DR

The paper tackles the challenge of enhancing multimodal math reasoning in MLLMs by directly absorbing capabilities from off-the-shelf Math LLMs without tuning. It identifies two key hurdles—finding math-reasoning–related parameters and aligning the parameter spaces of LLMs and MLLMs—and introduces IP-Merging, a tune-free method that identifies crucial layers via subspace similarity, rescales them, and projects them into the MLLM subspace before merging. Empirical results across MathVista, MathVerse, DynaMath, MathVision, and general QA benchmarks show consistent math reasoning gains with minimal disruption to non-math abilities, and ablations confirm the complementary benefits of selection, rescaling, and projection. The approach scales to larger and multiple models and demonstrates transferability across System-1 and System-2 LLMs, suggesting practical utility for augmenting multimodal reasoning without expensive data collection or retraining.

Abstract

Math reasoning has been one crucial ability of large language models (LLMs), where significant advancements have been achieved in recent years. However, most efforts focus on LLMs by curating high-quality annotation data and intricate training (or inference) paradigms, while the math reasoning performance of multi-modal LLMs (MLLMs) remains lagging behind. Since the MLLM typically consists of an LLM and a vision block, we wonder: Can MLLMs directly absorb math reasoning abilities from off-the-shelf math LLMs without tuning? Recent model-merging approaches may offer insights into this question. However, they overlook the alignment between the MLLM and LLM, where we find that there is a large gap between their parameter spaces, resulting in lower performance. Our empirical evidence reveals two key factors behind this issue: the identification of crucial reasoning-associated layers in the model and the mitigation of the gaps in parameter space. Based on the empirical insights, we propose IP-Merging that first identifies the reasoning-associated parameters in both MLLM and Math LLM, then projects them into the subspace of MLLM, aiming to maintain the alignment, and finally merges parameters in this subspace. IP-Merging is a tuning-free approach since parameters are directly adjusted. Extensive experiments demonstrate that our IP-Merging method can enhance the math reasoning ability of MLLMs directly from Math LLMs without compromising their other capabilities.
Paper Structure (35 sections, 1 theorem, 13 equations, 8 figures, 10 tables, 1 algorithm)

This paper contains 35 sections, 1 theorem, 13 equations, 8 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Math Reasoning of MLLMs
  4. Model Merging
  5. Model Merging Framework
  6. Preliminaries of Task Vectors
  7. Model Merging Framework
  8. Methodology
  9. Identify Math-Reasoning-Associated Parameters
  10. Gaps between Models in Parameter Space
  11. IP-Merging
  12. Parameter Identification
  13. Parameter Projection
  14. Experiments
  15. Experiments Setup
  16. ...and 20 more sections

Key Result

Proposition 1

Let $\Delta\boldsymbol{W}_{Math}$ and $\Delta\boldsymbol{W}_{MLLM}$, be two real matrices. The L2-norm of two matrices is defined by $\| \Delta\boldsymbol{W}_{Math} \|_2 =\sigma_{\max}(\Delta\boldsymbol{W}_{Math}), \| \Delta\boldsymbol{W}_{MLLM} \|_2=\sigma_{\max}(\Delta\boldsymbol{W}_{MLLM}),$ Mod If we assume then,

Figures (8)

  • Figure 1: We aim to enhance multi-modal reasoning skills of MLLM by merging parameters in MLLM and Math LLM without tuning or changing the size of the model.
  • Figure 3: (a) Absolute cosine value of the top 100 corresponding angles in one MLP layer. (b) The highest similarity score distribution among model parameters, orange bars represent the parameters with a high similarity score.
  • Figure 4: (a) Singular values distribution in one MLP layer. We plot the top 1024 singular values. (b) Maximum singular values across all model parameters.
  • Figure 5: Selected layers in the LLaVA model for merging.
  • Figure 6: Case study of examples are correctly answered after merging the math LLM.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Definition 1: Similarity Value of Corresponding Angle
  • Proposition 1