Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process

TL;DR

The paper tackles multimodal fusion by balancing intra-modal feature preservation with cross-modal alignment, addressing missing modalities. It introduces DPMM, a DP-driven Gaussian mixture framework where mixture weights are allocated via a Dirichlet process, enabling amplification of prominent intra-modal components and implicit cross-modal alignment. Estimation is performed with stochastic variational inference, and missing modalities can be imputed from learned marginals to produce informative embeddings. Empirical results on MIMIC-III/IV and CMU-MOSI/POM show consistent improvements over strong baselines, with ablations confirming the gains stem from both feature amplification and alignment, illustrating a principled approach to robust multimodal fusion in the presence of incomplete data.

Abstract

Developing effective multimodal fusion approaches has become increasingly essential in many real-world scenarios, such as health care and finance. The key challenge is how to preserve the feature expressiveness in each modality while learning cross-modal interactions. Previous approaches primarily focus on the cross-modal alignment, while over-emphasis on the alignment of marginal distributions of modalities may impose excess regularization and obstruct meaningful representations within each modality. The Dirichlet process (DP) mixture model is a powerful Bayesian non-parametric method that can amplify the most prominent features by its richer-gets-richer property, which allocates increasing weights to them. Inspired by this unique characteristic of DP, we propose a new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment. Specifically, we assume that each modality follows a mixture of multivariate Gaussian distributions and further adopt DP to calculate the mixture weights for all the components. This paradigm allows DP to dynamically allocate the contributions of features and select the most prominent ones, leveraging its richer-gets-richer property, thus facilitating multimodal feature fusion. Extensive experiments on several multimodal datasets demonstrate the superior performance of our model over other competitors. Ablation analysis further validates the effectiveness of DP in aligning modality distributions and its robustness to changes in key hyperparameters. Code is anonymously available at https://github.com/HKU-MedAI/DPMM.git

Figures (6)

• Figure 1: Dirichlet process based modality alignment (top right) vs. conventional modality alignment (bottom right). The Dirichlet process allocates more weights to capture the prominent features (e.g., $\mathbin{\hbox{$\bigstar$}},\triangle$) while conventional methods inevitably shrink the intra-modal features during alignment.
• Figure 2: Overview of our proposed DPMM framework. We first input the data to the encoders to obtain latent embeddings $\bm z_1, \ldots, \bm z_M$. We then compute the mixture weights $\{\pi_{mk}: m \in [M], k \in [K]\}$ using the stick-breaking formulation of the DP. The weighted likelihood is then computed with the mixture weights and the multivariate Gaussian mixture assumption. Finally, by integrating the task-specific loss, the evidence lower bound (ELBO) for training the encoders can be obtained. To handle unpaired samples during training, we adopt gradient-preserving sampling which can impute the missing observations while the gradient can still be backpropagated.
• Figure 3: Illustration of the stick-breaking process under the multimodal context. We perform splitting by modality first and then by mixture components. This allows the DP to allocate weights to the most prominent features from each modality.
• Figure 4: Plate notation of our proposed DPMM model.
• Figure 5: Histogram of the log-density values of the top 2 mixtures of each of the $M$ modalities. We observe that DP can rank the feature components of each modality according to its prominence.

Theorems & Definitions (1)

• Theorem 1