Hellinger Multimodal Variational Autoencoders

TL;DR

HELVAE introduces a probabilistic opinion-pooling perspective for multimodal VAEs by adopting Hölder pooling with ${\alpha=0.5}$, which corresponds to the Hellinger distance. The authors derive a moment-matching aggregation that projects the pooled posterior onto a diagonal Gaussian, enabling efficient training without sub-sampling. The approach yields a new multimodal VAE that achieves stronger latent representations and better trade-offs between generative coherence and quality across benchmarks (PolyMNIST, CUB Image-Captions, bimodal CelebA), outperforming several state-of-the-art methods. They also extend to a Mixture of HELVAEs (MoHELVAE) over modality subsets, further enhancing performance in some settings, while maintaining computational efficiency compared to prior mixture-based models.

Abstract

Multimodal variational autoencoders (VAEs) are widely used for weakly supervised generative learning with multiple modalities. Predominant methods aggregate unimodal inference distributions using either a product of experts (PoE), a mixture of experts (MoE), or their combinations to approximate the joint posterior. In this work, we revisit multimodal inference through the lens of probabilistic opinion pooling, an optimization-based approach. We start from Hölder pooling with $α=0.5$, which corresponds to the unique symmetric member of the $α\text{-divergence}$ family, and derive a moment-matching approximation, termed Hellinger. We then leverage such an approximation to propose HELVAE, a multimodal VAE that avoids sub-sampling, yielding an efficient yet effective model that: (i) learns more expressive latent representations as additional modalities are observed; and (ii) empirically achieves better trade-offs between generative coherence and quality, outperforming state-of-the-art multimodal VAE models.

Figures (15)

• Figure 1: PoE, MoE, and Hölder pooling ($\alpha=0.5$) with two agreeing experts and one disagreeing sharp expert.
• Figure 2: Performance of PoE, MoE, Hölder pooling ($\alpha=0.5$), and Hellinger aggregators as a function of the number of good experts (with two bad experts fixed). Evaluation is based on negative log-likelihood ($\downarrow$), the Bhattacharyya coefficient ($\uparrow$), and sharpness ($\downarrow$). We show the minimum and maximum NLL values for PoE.
• Figure 3: Trade-offs on the PolyMNIST dataset between generative coherence ($\uparrow$) and log-likelihood estimation ($\uparrow$), as well as between generative coherence and generative quality ($\downarrow$), for $\beta \in \{1, 2.5, 5, 10\}$. For each model, the Pareto front (dashed line) connects the non-dominated points that achieve the best trade-offs.
• Figure 4: Conditionally generated samples of the second modality (fifth to last rows), given the corresponding test examples from the other four modalities (first four rows) on PolyMNIST, with each model evaluated at its best $\beta \in \{1, 2.5, 5, 10\}$.
• Figure 5: Conditional generative quality ($\downarrow$) on the PolyMNIST dataset for target modalities $X_1 \dots X_5$ as a function of the number of input modalities, averaged across subsets of a given size excluding the target. Markers indicate means, and shaded regions represent standard deviations.