Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisit Modality Imbalance at the Decision Layer

Xiaoyu Ma, Hao Chen

TL;DR

The paper investigates modality imbalance in multimodal learning, revealing that decision-layer biases toward certain modalities persist even after extensive pretraining. Through experiments on audio-visual datasets CREMAD and Kinetic-Sounds, it demonstrates that decision-layer weights and logits are skewed due to intrinsic differences in modality distributions, not solely optimization dynamics. It argues that aggregating uncalibrated modality outputs at the fusion stage yields biased decision weights and proposes adaptive, category-aware weight allocation at the decision layer. This approach aims to enable weaker modalities to contribute effectively by aligning decision weights with modality capabilities at the task level, with significant implications for more robust multimodal fusion.

Abstract

Multimodal learning integrates information from different modalities to enhance model performance, yet it often suffers from modality imbalance, where dominant modalities overshadow weaker ones during joint optimization. This paper reveals that such an imbalance not only occurs during representation learning but also manifests significantly at the decision layer. Experiments on audio-visual datasets (CREMAD and Kinetic-Sounds) show that even after extensive pretraining and balanced optimization, models still exhibit systematic bias toward certain modalities, such as audio. Further analysis demonstrates that this bias originates from intrinsic disparities in feature-space and decision-weight distributions rather than from optimization dynamics alone. We argue that aggregating uncalibrated modality outputs at the fusion stage leads to biased decision-layer weighting, hindering weaker modalities from contributing effectively. To address this, we propose that future multimodal systems should focus more on incorporate adaptive weight allocation mechanisms at the decision layer, enabling relative balanced according to the capabilities of each modality.

Revisit Modality Imbalance at the Decision Layer

TL;DR

The paper investigates modality imbalance in multimodal learning, revealing that decision-layer biases toward certain modalities persist even after extensive pretraining. Through experiments on audio-visual datasets CREMAD and Kinetic-Sounds, it demonstrates that decision-layer weights and logits are skewed due to intrinsic differences in modality distributions, not solely optimization dynamics. It argues that aggregating uncalibrated modality outputs at the fusion stage yields biased decision weights and proposes adaptive, category-aware weight allocation at the decision layer. This approach aims to enable weaker modalities to contribute effectively by aligning decision weights with modality capabilities at the task level, with significant implications for more robust multimodal fusion.

Abstract

Multimodal learning integrates information from different modalities to enhance model performance, yet it often suffers from modality imbalance, where dominant modalities overshadow weaker ones during joint optimization. This paper reveals that such an imbalance not only occurs during representation learning but also manifests significantly at the decision layer. Experiments on audio-visual datasets (CREMAD and Kinetic-Sounds) show that even after extensive pretraining and balanced optimization, models still exhibit systematic bias toward certain modalities, such as audio. Further analysis demonstrates that this bias originates from intrinsic disparities in feature-space and decision-weight distributions rather than from optimization dynamics alone. We argue that aggregating uncalibrated modality outputs at the fusion stage leads to biased decision-layer weighting, hindering weaker modalities from contributing effectively. To address this, we propose that future multimodal systems should focus more on incorporate adaptive weight allocation mechanisms at the decision layer, enabling relative balanced according to the capabilities of each modality.
Paper Structure (6 sections, 4 figures, 1 table)

This paper contains 6 sections, 4 figures, 1 table.

Table of Contents

  1. Balanced Multimodal Learning
  2. Observation of Modality Imbalance at Decision Layer
  3. Analysis of Modality Imbalance at Decision Layer
  4. Possible Causes of Modality Imbalance at the Decision Layer
  5. Should We Pursue Decision-Layer Balance?
  6. Conclusion

Figures (4)

  • Figure 1: Heatmap of decision-layer weights for a joint-trained multimodal model. The model employs concatenation for feature fusion, with the first 512 dimensions corresponding to audio features and the last 512 dimensions corresponding to video features.
  • Figure 2: Heatmap of decision-layer weights for a pretrain-based multimodal model. The model employs concatenation for feature fusion, with the first 512 dimensions corresponding to audio features and the last 512 dimensions corresponding to video features.
  • Figure 3: Distribution of decision-layer weights for different unimodal models trained on audiovisual datasets.
  • Figure 4: Per-Category Accuracy and Modality Weight at the Decision Layer. The bar chart represents the unimodal prediction accuracy of each modality across categories, while the line chart illustrates the average L1-norm of decision-layer weights for each modality in different categories.