Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts

Ruofeng Yang, Yongcan Li, Bo Jiang, Cheng Chen, Shuai Li

TL;DR

This paper addresses why diffusion models can learn effectively from small datasets by modeling data as a union of low-dimensional, multi-modal manifolds. It introduces Mixture of Low-Rank MoG (MoLR-MoG) modeling, which imposes a subspace structure with a mixture of Gaussians in each subspace, resulting in a MoE-style nonlinear score function. The authors provide theoretical guarantees: an estimation error bound that scales with subspace count and latent dimensions, and a local strong convexity-based convergence guarantee for gradient-based optimization, even under multi-modal and partially overlapping conditions. Empirically, MoLR-MoG with MoE-latent MoG scores yields competitive generation quality with an order-of-magnitude reduction in parameters compared to Unet baselines and explains fast optimization observed in diffusion models, bridging theory and practice.

Abstract

Recently, diffusion models have achieved a great performance with a small dataset of size $n$ and a fast optimization process. However, the estimation error of diffusion models suffers from the curse of dimensionality $n^{-1/D}$ with the data dimension $D$. Since images are usually a union of low-dimensional manifolds, current works model the data as a union of linear subspaces with Gaussian latent and achieve a $1/\sqrt{n}$ bound. Though this modeling reflects the multi-manifold property, the Gaussian latent can not capture the multi-modal property of the latent manifold. To bridge this gap, we propose the mixture subspace of low-rank mixture of Gaussian (MoLR-MoG) modeling, which models the target data as a union of $K$ linear subspaces, and each subspace admits a mixture of Gaussian latent ($n_k$ modals with dimension $d_k$). With this modeling, the corresponding score function naturally has a mixture of expert (MoE) structure, captures the multi-modal information, and contains nonlinear property. We first conduct real-world experiments to show that the generation results of MoE-latent MoG NN are much better than MoE-latent Gaussian score. Furthermore, MoE-latent MoG NN achieves a comparable performance with MoE-latent Unet with $10 \times$ parameters. These results indicate that the MoLR-MoG modeling is reasonable and suitable for real-world data. After that, based on such MoE-latent MoG score, we provide a $R^4\sqrt{Σ_{k=1}^Kn_k}\sqrt{Σ_{k=1}^Kn_kd_k}/\sqrt{n}$ estimation error, which escapes the curse of dimensionality by using data structure. Finally, we study the optimization process and prove the convergence guarantee under the MoLR-MoG modeling. Combined with these results, under a setting close to real-world data, this work explains why diffusion models only require a small training sample and enjoy a fast optimization process to achieve a great performance.

Multi-Subspace Multi-Modal Modeling for Diffusion Models: Estimation, Convergence and Mixture of Experts

TL;DR

This paper addresses why diffusion models can learn effectively from small datasets by modeling data as a union of low-dimensional, multi-modal manifolds. It introduces Mixture of Low-Rank MoG (MoLR-MoG) modeling, which imposes a subspace structure with a mixture of Gaussians in each subspace, resulting in a MoE-style nonlinear score function. The authors provide theoretical guarantees: an estimation error bound that scales with subspace count and latent dimensions, and a local strong convexity-based convergence guarantee for gradient-based optimization, even under multi-modal and partially overlapping conditions. Empirically, MoLR-MoG with MoE-latent MoG scores yields competitive generation quality with an order-of-magnitude reduction in parameters compared to Unet baselines and explains fast optimization observed in diffusion models, bridging theory and practice.

Abstract

Recently, diffusion models have achieved a great performance with a small dataset of size and a fast optimization process. However, the estimation error of diffusion models suffers from the curse of dimensionality with the data dimension . Since images are usually a union of low-dimensional manifolds, current works model the data as a union of linear subspaces with Gaussian latent and achieve a bound. Though this modeling reflects the multi-manifold property, the Gaussian latent can not capture the multi-modal property of the latent manifold. To bridge this gap, we propose the mixture subspace of low-rank mixture of Gaussian (MoLR-MoG) modeling, which models the target data as a union of linear subspaces, and each subspace admits a mixture of Gaussian latent ( modals with dimension ). With this modeling, the corresponding score function naturally has a mixture of expert (MoE) structure, captures the multi-modal information, and contains nonlinear property. We first conduct real-world experiments to show that the generation results of MoE-latent MoG NN are much better than MoE-latent Gaussian score. Furthermore, MoE-latent MoG NN achieves a comparable performance with MoE-latent Unet with parameters. These results indicate that the MoLR-MoG modeling is reasonable and suitable for real-world data. After that, based on such MoE-latent MoG score, we provide a estimation error, which escapes the curse of dimensionality by using data structure. Finally, we study the optimization process and prove the convergence guarantee under the MoLR-MoG modeling. Combined with these results, under a setting close to real-world data, this work explains why diffusion models only require a small training sample and enjoy a fast optimization process to achieve a great performance.
Paper Structure (39 sections, 21 theorems, 231 equations, 5 figures)

This paper contains 39 sections, 21 theorems, 231 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related Work
  4. Preliminaries
  5. Mixture of low-rank mixture of Gaussian (MoLR-MoG) Modeling
  6. Mixture of Experts (MoE)-Nonlinear MoG score.
  7. Experiments for MoE-latent MoG Score
  8. Discussion.
  9. Discussion on Expert-Specific VAE.
  10. Escape the Curse of Dimensionality With MoLR-MoG Modeling
  11. Strongly Convex Property and Convergence Guarantee
  12. 2-Modal Latent MoG Hessian Analysis and Optimization
  13. General MoG Latent Hessian Analysis and Optimization
  14. Analysis Without Highly Separated Condition
  15. Conclusion
  16. ...and 24 more sections

Key Result

Lemma 5.1

[Lipschitz Continuity] Let $L_{\mu_l}$ and $L_{U_k}$ be the Lipschitz constant w.r.t. $s_{\theta}$. With MoLR-MoG modeling and x in D, there is a constant such that for any $\theta,\theta'$, $\bigl\|s_\theta(x,t)-s_{\theta'}(x,t)\bigr\|_2\;\le\;L\,\|\theta-\theta'\|_2$, where $C_w=\frac{(R+s_tB_\mu)^3s_t^2}{\gamma_t^4}$,$B_\mu=\underset{k,l}{\max}\|\mu_{k,l}\|_2$. For $s_{\theta}$ and $s^*$, we h

Figures (5)

  • Figure 1: ImageNet results with expert specific VAE and small latent 2-layer Softmax-type network.
  • Figure 2: MoLR-MoG Modeling and Corresponding Nonlinear Score
  • Figure 3: Results of Different Modeling on Real-world Data.
  • Figure 4: Loss Curve for CIFAR-10
  • Figure 5: MoLR-MoG with Different VAE

Theorems & Definitions (36)

  • Remark 3.1: Comparison with MoLRG modeling
  • Lemma 5.1
  • Theorem 5.2
  • Lemma 6.0
  • Theorem 6.1
  • Corollary 6.1
  • Remark 6.2: Separated Gaussian Simplification
  • Lemma 6.2
  • Lemma 6.2
  • Remark 6.3: Previous MoG Learning through Score Matching
  • ...and 26 more