Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Negative Learning

Baoquan Gong, Xiyuan Gao, Pengfei Zhu, Qinghua Hu, Bing Cao

TL;DR

The paper tackles modality imbalance in multimodal learning by proposing Multimodal Negative Learning (MNL), a negative-learning paradigm where robust dominant modalities guide weaker ones to suppress non-target classes rather than aligning to target-class predictions. It introduces Unimodal Confidence Margin (UCoM) and derives a robustness lower bound for decision-level fusion, showing that increasing the weaker modalities' margins improves stability. MNL formalizes a dynamic guidance mechanism that selects which modality should guide another based on confidence and UCoM, and augments the loss with a targeted non-target class suppression term while keeping standard cross-entropy on target classes. Empirical results across multiple datasets demonstrate improved robustness and reduced empirical error for weak modalities, particularly under noise and imbalance, with compatibility across static late fusion and several dynamic fusion baselines. The work provides a theoretically grounded, practically flexible approach to robust multimodal fusion and suggests extensions to larger-scale LLM/MLLM fusion and more complex multimodal settings.

Abstract

Multimodal learning systems often encounter challenges related to modality imbalance, where a dominant modality may overshadow others, thereby hindering the learning of weak modalities. Conventional approaches often force weak modalities to align with dominant ones in "Learning to be (the same)" (Positive Learning), which risks suppressing the unique information inherent in the weak modalities. To address this challenge, we offer a new learning paradigm: "Learning Not to be" (Negative Learning). Instead of enhancing weak modalities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specific information, allowing weak modalities to preserve unique information without being over-aligned. We proceed to reveal multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error of weak modalities, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of our approach against competing methods. The code will be available at https://github.com/BaoquanGong/Multimodal-Negative-Learning.git.

Multimodal Negative Learning

TL;DR

The paper tackles modality imbalance in multimodal learning by proposing Multimodal Negative Learning (MNL), a negative-learning paradigm where robust dominant modalities guide weaker ones to suppress non-target classes rather than aligning to target-class predictions. It introduces Unimodal Confidence Margin (UCoM) and derives a robustness lower bound for decision-level fusion, showing that increasing the weaker modalities' margins improves stability. MNL formalizes a dynamic guidance mechanism that selects which modality should guide another based on confidence and UCoM, and augments the loss with a targeted non-target class suppression term while keeping standard cross-entropy on target classes. Empirical results across multiple datasets demonstrate improved robustness and reduced empirical error for weak modalities, particularly under noise and imbalance, with compatibility across static late fusion and several dynamic fusion baselines. The work provides a theoretically grounded, practically flexible approach to robust multimodal fusion and suggests extensions to larger-scale LLM/MLLM fusion and more complex multimodal settings.

Abstract

Multimodal learning systems often encounter challenges related to modality imbalance, where a dominant modality may overshadow others, thereby hindering the learning of weak modalities. Conventional approaches often force weak modalities to align with dominant ones in "Learning to be (the same)" (Positive Learning), which risks suppressing the unique information inherent in the weak modalities. To address this challenge, we offer a new learning paradigm: "Learning Not to be" (Negative Learning). Instead of enhancing weak modalities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specific information, allowing weak modalities to preserve unique information without being over-aligned. We proceed to reveal multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error of weak modalities, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of our approach against competing methods. The code will be available at https://github.com/BaoquanGong/Multimodal-Negative-Learning.git.
Paper Structure (35 sections, 21 equations, 8 figures, 14 tables, 1 algorithm)

This paper contains 35 sections, 21 equations, 8 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Imbalanced Multimodal Learning
  4. Robustness in Multimodal Cooperation
  5. Method
  6. Basic Setting
  7. Robust Lower Bound for Multimodal Learning with Late Fusion
  8. Multimodal Negative Learning
  9. Dynamic Guidance and Training Strategy
  10. Experiments
  11. Experimental Setup
  12. Results
  13. Ablation Study
  14. Discussion
  15. Conclusion
  16. ...and 20 more sections

Figures (8)

  • Figure 1: (a) Illustration of conventional (top) vs. our (bottom) strategies: instead of improving target prediction of weak modalities, we stabilize their decision space by suppressing non-target classes, which can preserve the modality-specific information and enhance robustness. (b) Empirical evidence: forced alignment can degrade weak modality (Video) predictions on certain samples, highlighting the risk of losing modality diversity. Where KL Guidance stands for forced alignment method.
  • Figure 2: After cross entropy optimization, each unimodal learner updates its parameters based on ground-truth supervision, overlooking non-target class optimization, especially problematic for inferior modalities, which introduce noise during fusion. With the MNL, the robust dominant modality guides the inferior one by reducing uncertainty over non-target classes, thereby enhancing multimodal robustness with a larger UCoM.
  • Figure 3: Accuracy varies with Gaussian noise level on the CREMA-D dataset. “+” indicates models trained with the proposed Multimodal Negative Learning (MNL). Our method consistently improves accuracy across all noise levels.
  • Figure 4: (a) We visualize the average KL divergence between the two modality predictions and compare the performance gaps between our method and the baseline across varying noise levels. (b) and (c) show the evolution of UCoM on the CREMA-D and MVSA datasets, respectively.
  • Figure 5: During the training process, examples are provided for both the guiding and guided modalities, with the green bar representing the ground truth class. Specifically, (d) highlights the predicted probabilities for the ground truth class and the runner-up class.
  • ...and 3 more figures