Mixup Helps Understanding Multimodal Video Better

TL;DR

The paper tackles overfitting to dominant modalities in multimodal video understanding by introducing Multimodal Mixup (MM), which applies mixup at the aggregated feature level to generate virtual samples and improve generalization. Building on MM, Balanced Multimodal Mixup (B-MM) dynamically adjusts per-modality mixing based on each modality's learning contribution, addressing modality imbalance. Experiments on CREMAD, Kinetic-Sounds, and UCF-101 reveal consistent gains over traditional fusion and existing balanced learning methods, improving both emotion and action recognition performance. These results demonstrate that adaptive, modality-aware data augmentation can enhance robustness and cross-modal cooperation in multimodal video tasks, with implications for broader multimodal learning settings.

Abstract

Multimodal video understanding plays a crucial role in tasks such as action recognition and emotion classification by combining information from different modalities. However, multimodal models are prone to overfitting strong modalities, which can dominate learning and suppress the contributions of weaker ones. To address this challenge, we first propose Multimodal Mixup (MM), which applies the Mixup strategy at the aggregated multimodal feature level to mitigate overfitting by generating virtual feature-label pairs. While MM effectively improves generalization, it treats all modalities uniformly and does not account for modality imbalance during training. Building on MM, we further introduce Balanced Multimodal Mixup (B-MM), which dynamically adjusts the mixing ratios for each modality based on their relative contributions to the learning objective. Extensive experiments on several datasets demonstrate the effectiveness of our methods in improving generalization and multimodal robustness.

Figures (5)

• Figure 1: Training and test accuracy curves of the multimodal model and its individual branches during the learning process on the CREMAD dataset.
• Figure 2: The pipeline of the Multimodal Mixup (MM) method. For each batch, the feature representations of individual modal inputs are first extracted and fused to obtain a multimodal feature representation. The mixup mixup method is then applied to the multimodal feature representation with $\lambda$ as the mixing parameter to generate virtual feature-label pairs, which are used for model learning.
• Figure 3: The pipeline of Balanced Multimodal Mixup (B-MM) method. Similar to the MM method, we first get the feature representations of individual modal inputs. Then we apply the mixup method to the unimodal features according to the parameters $\lambda^a$ and $\lambda^v$. After each epoch, the two parameters will update according to the discrepancy ratio $\rho$peng2022balanced of modalities.
• Figure 4: UMAP visualizations of feature representations from the Baseline models on the training and test sets. Within each configuration, visualizations for audio, video, and multimodal features are included. Different colors indicate different classes.
• Figure 5: UMAP visualizations of feature representations from the B-MM models on the training and test sets. Within each configuration, visualizations for audio, video, and multimodal features are provided. Different colors indicate different classes.