Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes

Guanyu Yao, Qiucheng Wu, Yang Zhang, Zhaowen Wang, Handong Zhao, Shiyu Chang

TL;DR

This work identifies a modality gap in multimodal LLMs where textual reasoning dominates visual understanding and shows that standard training often widens this gap. It analyzes the gap using datasets PGPS9K and MathVerse and proposes two RL-based remedies: data-level strategies (mixed and curriculum data) and a loss-level strategy (contrastive KL self-distillation) to preserve textual reasoning while boosting visual grounding. Empirical results demonstrate that curriculum training generally outperforms naive data mixing and that the KL objective improves in-distribution performance, with some cross-domain limitations due to dataset annotation differences. The findings offer a practical training recipe—curriculum training plus a KL-based self-distillation loss—to produce more balanced MLLMs with stronger visual reasoning capabilities, while acknowledging dataset-specific limitations and suggesting future work on more diverse benchmarks and models.

Abstract

Multimodal large language models (MLLMs) have demonstrated strong capabilities on vision-and-language tasks. However, recent findings reveal an imbalance in their reasoning capabilities across visual and textual modalities. Specifically, current MLLMs often over-rely on textual cues while under-attending to visual content, resulting in suboptimal performance on tasks that require genuine visual reasoning. We refer to this phenomenon as the \textit{modality gap}, defined as the performance disparity between text-centric and vision-centric inputs. In this paper, we analyze the modality gap through the lens of training recipes. We first show that existing training recipes tend to amplify this gap. Then, we systematically explore strategies to bridge it from two complementary perspectives: data and loss design. Our findings provide insights into developing training recipes that mitigate the modality gap and promote more balanced multimodal reasoning. Our code is publicly available at https://github.com/UCSB-NLP-Chang/Bridging-Modality-Gap.

Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes

TL;DR

This work identifies a modality gap in multimodal LLMs where textual reasoning dominates visual understanding and shows that standard training often widens this gap. It analyzes the gap using datasets PGPS9K and MathVerse and proposes two RL-based remedies: data-level strategies (mixed and curriculum data) and a loss-level strategy (contrastive KL self-distillation) to preserve textual reasoning while boosting visual grounding. Empirical results demonstrate that curriculum training generally outperforms naive data mixing and that the KL objective improves in-distribution performance, with some cross-domain limitations due to dataset annotation differences. The findings offer a practical training recipe—curriculum training plus a KL-based self-distillation loss—to produce more balanced MLLMs with stronger visual reasoning capabilities, while acknowledging dataset-specific limitations and suggesting future work on more diverse benchmarks and models.

Abstract

Multimodal large language models (MLLMs) have demonstrated strong capabilities on vision-and-language tasks. However, recent findings reveal an imbalance in their reasoning capabilities across visual and textual modalities. Specifically, current MLLMs often over-rely on textual cues while under-attending to visual content, resulting in suboptimal performance on tasks that require genuine visual reasoning. We refer to this phenomenon as the \textit{modality gap}, defined as the performance disparity between text-centric and vision-centric inputs. In this paper, we analyze the modality gap through the lens of training recipes. We first show that existing training recipes tend to amplify this gap. Then, we systematically explore strategies to bridge it from two complementary perspectives: data and loss design. Our findings provide insights into developing training recipes that mitigate the modality gap and promote more balanced multimodal reasoning. Our code is publicly available at https://github.com/UCSB-NLP-Chang/Bridging-Modality-Gap.
Paper Structure (24 sections, 1 equation, 4 figures, 4 tables)

This paper contains 24 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Visual Reasoning in MLLMs
  5. Modality Gap in MLLMs
  6. Metrics.
  7. Direct Inference Results.
  8. Effect of Standard RL Training.
  9. Mitigating Modality Gap
  10. Data Perspective
  11. Implementation.
  12. Result.
  13. Loss Perspective
  14. Implementation.
  15. Result.
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Current MLLMs exhibit an imbalance between visual and textual reasoning. When information present in the visual modality is removed from the text, the MLLM fails to answer the question.
  • Figure 2: Standard training recipe widens modality gap.
  • Figure 3: We consider two types of data: ❶ both text and image contain complete information, referred to as full text; and ❷ the text omits information already present in the image, referred to as partial text. We then analyze better training recipe from both data and loss perspectives.
  • Figure 4: Annotation style mismatch between PGPS9K and MathVerse. PGPS9K diagrams explicitly mark key geometric relations—parallelism and equality of segments/angles—whereas MathVerse omits such markings. Models trained on PGPS9K may over-rely on these visual tags and struggle to infer relations on MathVerse, weakening out-of-distribution generalization.