MLLMs are Deeply Affected by Modality Bias

TL;DR

Modality bias in Multimodal Large Language Models (MLLMs) arises when language priors dominate learning, hindering effective use of visual and other modalities. The authors formalize modality bias, propose a three-direction research roadmap, and identify five contributing factors (three primary, two secondary), supported by a case study on MMMU-Pro with Qwen2.5VL. They offer targeted solutions spanning dataset design, balanced training strategies, and preference-based debiasing, and advocate for explainable AI to uncover underlying mechanisms. This work highlights the need for balanced cross-modal integration to advance robust, generalizable multimodal systems and progress toward Artificial General Intelligence.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have shown promising results in integrating diverse modalities such as texts and images. MLLMs are heavily influenced by modality bias, often relying on language while under-utilizing other modalities like visual inputs. This position paper argues that MLLMs are deeply affected by modality bias. Firstly, we diagnose the current state of modality bias, highlighting its manifestations across various tasks. Secondly, we propose a systematic research road-map related to modality bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and offer actionable suggestions for future research to mitigate it. To substantiate these findings, we conduct experiments that demonstrate the influence of each factor: 1. Data Characteristics: Language data is compact and abstract, while visual data is redundant and complex, creating an inherent imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The dominance of pretrained language models in MLLMs leads to overreliance on language and neglect of visual information. 3. Training Objectives: Current objectives often fail to promote balanced cross-modal alignment, resulting in shortcut learning biased toward language. These findings highlight the need for balanced training strategies and model architectures to better integrate multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle these challenges and drive innovation in MLLM research. Our work provides a fresh perspective on modality bias in MLLMs and offers insights for developing more robust and generalizable multimodal systems-advancing progress toward Artificial General Intelligence.

Figures (4)

• Figure 1: The big picture of our position on modality bias in MLLMs. (a) We define modality bias in multimodal models, with a focus on MLLMs. (b) We outline a research roadmap on modality bias, highlighting three key directions. (c) We summarize five contributing factors to modality bias: three primary and two secondary.
• Figure 2: Further definition of modality bias and three potential results.
• Figure 3: Case study for exploring modality bias in MLLMs. Dataset: MMMU-Pro, MLLM: Qwen2.5VL. Based on this case study, the three main factors proposed in Sec. \ref{['fac']} are further illustrated and proved. "white" means the image pixels are all set to 255. "black" means the image pixels are all set to 0.
• Figure 4: Targeted solutions of modality bias in MLLMs, including current works and future directions.