Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Multimodal Learning via Representation Decoupling

Shicai Wei, Yang Luo, Yuji Wang, Chunbo Luo

TL;DR

This paper tackles the challenge of incomplete multimodal learning by identifying an intra-class direction constraint in traditional common subspace methods and proposing a Decoupled Multimodal Representation Network (DMRNet). DMRNet replaces fixed embeddings with probabilistic embeddings over modality combinations, sampling training representations via reparameterization while using the mean embedding for inference, thereby relaxing inter-class directional constraints and enabling modality-specific information capture. A distributional regularizer and a hard combination regularizer further improve learning across hard modality combinations, with the model demonstrated to outperform state-of-the-art methods on CASIA-SURF, CREMA-D, Kinect-Sounds, and NYUv2 segmentation tasks. The approach yields improved robustness to missing modalities and extends to dense prediction settings, offering practical benefits for real-world multimodal systems.

Abstract

Multimodal learning robust to missing modality has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. However, we reveal that they are sub-optimal due to their implicit constraint on intra-class representation. Specifically, the sample with different modalities within the same class will be forced to learn representations in the same direction. This hinders the model from capturing modality-specific information, resulting in insufficient learning. To this end, we propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning. Specifically, DMRNet models the input from different modality combinations as a probabilistic distribution instead of a fixed point in the latent space, and samples embeddings from the distribution for the prediction module to calculate the task loss. As a result, the direction constraint from the loss minimization is blocked by the sampled representation. This relaxes the constraint on the inference representation and enables the model to capture the specific information for different modality combinations. Furthermore, we introduce a hard combination regularizer to prevent DMRNet from unbalanced training by guiding it to pay more attention to hard modality combinations. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that the proposed DMRNet outperforms the state-of-the-art significantly.

Robust Multimodal Learning via Representation Decoupling

TL;DR

This paper tackles the challenge of incomplete multimodal learning by identifying an intra-class direction constraint in traditional common subspace methods and proposing a Decoupled Multimodal Representation Network (DMRNet). DMRNet replaces fixed embeddings with probabilistic embeddings over modality combinations, sampling training representations via reparameterization while using the mean embedding for inference, thereby relaxing inter-class directional constraints and enabling modality-specific information capture. A distributional regularizer and a hard combination regularizer further improve learning across hard modality combinations, with the model demonstrated to outperform state-of-the-art methods on CASIA-SURF, CREMA-D, Kinect-Sounds, and NYUv2 segmentation tasks. The approach yields improved robustness to missing modalities and extends to dense prediction settings, offering practical benefits for real-world multimodal systems.

Abstract

Multimodal learning robust to missing modality has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. However, we reveal that they are sub-optimal due to their implicit constraint on intra-class representation. Specifically, the sample with different modalities within the same class will be forced to learn representations in the same direction. This hinders the model from capturing modality-specific information, resulting in insufficient learning. To this end, we propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning. Specifically, DMRNet models the input from different modality combinations as a probabilistic distribution instead of a fixed point in the latent space, and samples embeddings from the distribution for the prediction module to calculate the task loss. As a result, the direction constraint from the loss minimization is blocked by the sampled representation. This relaxes the constraint on the inference representation and enables the model to capture the specific information for different modality combinations. Furthermore, we introduce a hard combination regularizer to prevent DMRNet from unbalanced training by guiding it to pay more attention to hard modality combinations. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that the proposed DMRNet outperforms the state-of-the-art significantly.
Paper Structure (16 sections, 7 equations, 2 figures, 8 tables)

This paper contains 16 sections, 7 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Incomplete multimodal learning
  4. Probabilistic Embeddings
  5. Methods
  6. Analysis of Representation Constraint
  7. Decoupled Multimodal Representation
  8. Hard Combination Regularizer
  9. Total Loss
  10. Relationship to Prior Work
  11. Experiments
  12. Performance and Comparison on Face Anti-spoofing Task
  13. Performance and Comparison on Audio-Visual Recognition Task
  14. Ablation Study
  15. Parameters Analysis
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Illustration of the histogram of the inter-channel distance matrix $D_{channel}$ (defined in Section \ref{['abs']}) on the CASIA-SURF dataset. (a) and (b) show the intra-modality inter-channel distance between feature maps of the RGB encoder. (c) and (d) show the inter-modality inter-channel distance between the feature maps of RGB and Depth encoders. 'Vanilla' denotes the conventional multimodal model. 'Unimodal' means the baseline that trains models for each modality independently, which provides the ideal feature diversity without inter-modality interference. 'A:B' denotes the ratio of histogram mean. Higher inter-channel distance means higher diversity .
  • Figure 2: The overall framework of the proposed DMRNet. It consists of two parts: 1) the decoupled multimodal representation that decouples the inference representation and training representation to alleviate the direction constraint on the inference representation; and 2) the hard combination regularizer that mines and regularizes the hard modality combinations to handle the unbalanced training problem.