Multi-Component VAE with Gaussian Markov Random Field

Abstract

Multi-component datasets with intricate dependencies, like industrial assemblies or multi-modal imaging, challenge current generative modeling techniques. Existing Multi-component Variational AutoEncoders typically rely on simplified aggregation strategies, neglecting critical nuances and consequently compromising structural coherence across generated components. To explicitly address this gap, we introduce the Gaussian Markov Random Field Multi-Component Variational AutoEncoder , a novel generative framework embedding Gaussian Markov Random Fields into both prior and posterior distributions. This design choice explicitly models cross-component relationships, enabling richer representation and faithful reproduction of complex interactions. Empirically, our GMRF MCVAE achieves state-of-the-art performance on a synthetic Copula dataset specifically constructed to evaluate intricate component relationships, demonstrates competitive results on the PolyMNIST benchmark, and significantly enhances structural coherence on the real-world BIKED dataset. Our results indicate that the GMRF MCVAE is especially suited for practical applications demanding robust and realistic modeling of multi-component coherence

Figures (6)

• Figure 1: A general MRF-based Multi-Component VAE: each component is assigned its own encoder-decoder pair, where the encoder learns unary potentials $\psi_i$. A global encoder models pairwise potentials $\psi_{i,j}$ among components. Sampling $\mathbf{z}$ from this MRF-based latent space captures cross-component relationships. In practice, we adopt a Gaussian assumption for computational simplicity.
• Figure 2: Qualitative results for the unconditional generations on the Copula dataset. Each subplot visualizes joint distributions for each pair of coordinates $(\mathbf{x}_i^1, \mathbf{x}_j^1)$ and $(\mathbf{x}_i^2, \mathbf{x}_j^2)$ across the four two-dimensional components $(\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3, \mathbf{x}_4)$. The true distributions are depicted in orange and the generated ones in blue.
• Figure 3: PolyMNIST conditional generations. Each block corresponds to a model. In each column, the first image corresponds to the condition, followed by the conditionally generated components $M_i$.
• Figure 4: Conditional generation on BIKED. The first column shows the conditioning components, while each subsequent column presents the remaining generated components for each model, overlaid to form the complete bike.
• Figure 5: Qualitative analysis of unconditional generations using the Copula dataset. Each subplot displays the marginal distributions for each coordinate: $(\mathbf{x}_i^1)$ on the left and $(\mathbf{x}_i^2)$ on the right, across four two-dimensional components $(\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3, \mathbf{x}_4)$. True distributions are depicted in orange and generated distributions in blue.

Theorems & Definitions (5)

• Theorem 1
• proof
• Theorem 2
• Theorem 3
• proof