End-to-End Multi-Modal Diffusion Mamba

TL;DR

MDM introduces an end-to-end framework that unifies multi-modal processing through a VAE-based noisy latent encoder and a Mamba-powered multi-step diffusion decoder. It employs Score Entropy Loss to jointly optimize diffusion-based denoising and information selection across modalities, enabling simultaneous high-resolution image and long-text generation with improved efficiency. The approach achieves competitive or state-leading performance on image generation, captioning, VQA, and world-knowledge tasks, while reducing inference complexity to $\mathcal{O}(MLN^2)$ compared to traditional transformer-based architectures. Its DPM-Solver accelerated sampling further enhances practicality for long sequences and high-dimensional outputs, suggesting a promising direction for truly end-to-end multi-modal diffusion.</nobr>

Abstract

Current end-to-end multi-modal models utilize different encoders and decoders to process input and output information. This separation hinders the joint representation learning of various modalities. To unify multi-modal processing, we propose a novel architecture called MDM (Multi-modal Diffusion Mamba). MDM utilizes a Mamba-based multi-step selection diffusion model to progressively generate and refine modality-specific information through a unified variational autoencoder for both encoding and decoding. This innovative approach allows MDM to achieve superior performance when processing high-dimensional data, particularly in generating high-resolution images and extended text sequences simultaneously. Our evaluations in areas such as image generation, image captioning, visual question answering, text comprehension, and reasoning tasks demonstrate that MDM significantly outperforms existing end-to-end models (MonoFormer, LlamaGen, and Chameleon etc.) and competes effectively with SOTA models like GPT-4V, Gemini Pro, and Mistral. Our results validate MDM's effectiveness in unifying multi-modal processes while maintaining computational efficiency, establishing a new direction for end-to-end multi-modal architectures.

Figures (9)

• Figure 1: Comparison of three types of models.
• Figure 1: Image generation with CFG on ImageNet deng2009imagenet 256 $\times$ 256.
• Figure 2: Framework of Multi-Modal Diffusion Mamba. MDM first encodes inputs (caption, VQVAE-processed image, question) using VAE (a), while performing padding (class, diffusion timestep, token completion) and flatten operations (d, e). Next, data reconstruction is progressively completed via diffusion mamba operations (b), modeling images and text temporally through scanning processes (f, g) for efficient information selection (red boxes indicate selection). Selected data undergoes computation (i) guided by (h) within the Mamba-2 framework to update model parameters. Finally, the MDM output passes through the VAE decoder (c) to reconstruct real data.
• Figure 2: Image generation on COCO karpathy2015deep caption text.
• Figure 3: VQA, QA and Multi-Modal generation test from MDM. The results of VQA are part of VQAv2 goyal2017making. The QA results are part of PIQA bisk2020piqa and MMLU hendrycks2020measuring. The Multi-Modal generation results are tested with ground-truth data.

Theorems & Definitions (9)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof