Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces

Kevin Rojas, Yuchen Zhu, Sichen Zhu, Felix X. -F. Ye, Molei Tao

TL;DR

The paper addresses joint multimodal generation without collapsing modalities into a single unimodal representation by proposing a general multimodal diffusion framework that operates on arbitrary state spaces. It introduces decoupled time schedules per modality and a generalized score-matching objective to jointly learn across modalities, enabling unconditional and conditional generation within one model. The approach is demonstrated on text–image synthesis and mixed-type tabular data with encoder-free architectures that achieve competitive performance while remaining highly parameter-efficient. A noisy-guidance mechanism further enhances generation quality, and the work formalizes the theory behind decoupled-time multimodal diffusion, broadening diffusion applicability to diverse data types with minimal preprocessing.

Abstract

Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocessing protocols, such as tokenizers and variational autoencoders, to harmonize varied data representations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which can be problematic for applications with limited data. To lift this restriction, we propose a novel framework for building multimodal diffusion models on arbitrary state spaces, enabling native generation of coupled data across different modalities. By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously. We empirically validate our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance.

Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces

TL;DR

The paper addresses joint multimodal generation without collapsing modalities into a single unimodal representation by proposing a general multimodal diffusion framework that operates on arbitrary state spaces. It introduces decoupled time schedules per modality and a generalized score-matching objective to jointly learn across modalities, enabling unconditional and conditional generation within one model. The approach is demonstrated on text–image synthesis and mixed-type tabular data with encoder-free architectures that achieve competitive performance while remaining highly parameter-efficient. A noisy-guidance mechanism further enhances generation quality, and the work formalizes the theory behind decoupled-time multimodal diffusion, broadening diffusion applicability to diverse data types with minimal preprocessing.

Abstract

Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocessing protocols, such as tokenizers and variational autoencoders, to harmonize varied data representations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which can be problematic for applications with limited data. To lift this restriction, we propose a novel framework for building multimodal diffusion models on arbitrary state spaces, enabling native generation of coupled data across different modalities. By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously. We empirically validate our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance.

Paper Structure

This paper contains 43 sections, 10 theorems, 77 equations, 11 figures, 10 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Continuous Space Diffusion Models
  4. Discrete Space Diffusion Models
  5. Riemannian Diffusion Models
  6. Methodology
  7. Unified Perspective on Unimodal Diffusion Models
  8. Versatile Multimodal Diffusion Models with Decoupled Times
  9. Continuous-Discrete Multimodal Diffusion
  10. Noisy Guidance
  11. Experiments
  12. Text-Image Generation
  13. Mixed-type Tabular Data Synthesis
  14. More Information on Related Works
  15. Conclusions, Limitations and Future Works
  16. ...and 28 more sections

Key Result

Theorem 1

${\mathcal{I}}_{\mathrm{GESM}} \geq 0$, with equality reached when $\beta_\theta(\boldsymbol{x}, \boldsymbol{t}) \propto p(\boldsymbol{x}, \boldsymbol{t})$.

Figures (11)

  • Figure 1: By injecting noise into different modalities in a decoupled fashion, we enable the unconditional and modality-conditioned generation in a single model. (a) Joint generation of image and text. (b) Image generation given text captions as conditions. (c) Text generation given images as conditions.
  • Figure 2: Network backbone for text-image generation, motivated by MMDiT esser2024scaling and DiT peebles2023scalable.
  • Figure 3: Visualization of samples generated by our approach. Captions are truncated for brevity.
  • Figure 4: Performance of noisy guidance on MS-COCO FID-$10$K. We note that using partially noised conditions results in a better performance. A guidance interval of $t\in [0.3,0.8]$ was used.
  • Figure 5: Visual representation of ground truth labeled Riemannian data.
  • ...and 6 more figures

Theorems & Definitions (16)

  • Theorem 1
  • Theorem 2
  • Proposition 1
  • Proposition 2
  • proof
  • proof
  • proof
  • Lemma 1: Generator of the forward process
  • proof
  • Lemma 2: Generator of the backward process
  • ...and 6 more