Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion

TL;DR

CoM-DAD proposes a unified multimodal generation framework that bridges the discrete-continuous gap by coupling a continuous latent diffusion semantic planner with a discrete absorbing diffusion token generator. The method decouples high-level semantic planning from low-level token synthesis, using a Semantic Injection Interface and a Variable-Rate Noise Schedule, along with a Stochastic Mixed-Modal Transport strategy to align text and image modalities without heavy contrastive pre-training. Theoretical grounding is provided via an ELBO decomposition and two-stage diffusion processes, with experiments showing superior text fidelity, faster convergence, parallel decoding, and strong cross-modal alignment. This work offers a scalable path toward truly unified text-image generation, enabling zero-shot cross-modal capabilities with modest paired-data requirements and improved training stability compared to monolithic autoregressive or diffusion baselines.

Abstract

The bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders the development of truly unified multimodal systems. While Masked Language Models (MLMs) offer efficient bidirectional context, they traditionally lack the generative fidelity of autoregressive models and the semantic continuity of diffusion models. Furthermore, extending masked generation to multimodal settings introduces severe alignment challenges and training instability. In this work, we propose \textbf{CoM-DAD} (\textbf{Co}upled \textbf{M}anifold \textbf{D}iscrete \textbf{A}bsorbing \textbf{D}iffusion), a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process. CoM-DAD decouples high-level semantic planning from low-level token synthesis. First, we model the semantic manifold via a continuous latent diffusion process; second, we treat token generation as a discrete absorbing diffusion process, regulated by a \textbf{Variable-Rate Noise Schedule}, conditioned on these evolving semantic priors. Crucially, we introduce a \textbf{Stochastic Mixed-Modal Transport} strategy that aligns disparate modalities without requiring heavy contrastive dual-encoders. Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.

Figures (6)

• Figure 1: Overview of CoM-DAD. The framework splits generation into Macro-Planning (Top), where a continuous latent diffusion models abstract semantic "themes" (the "dreaming" phase), and Micro-Refinement (Bottom), where a discrete absorbing diffusion synthesizes tokens. The vertical arrow signifies the conditioning of the discrete generation process on the continuous prior, ensuring global alignment between the abstract plan and the final token sequence.
• Figure 2: The CoM-DAD Training Pipeline. The framework consists of two coupled diffusion processes. Left (Stage I): The Manifold-Constrained Semantic Diffusion learns a continuous prior over semantic representations ($r$) via an SDE, capable of handling both text and image modalities. Right (Stage II): The Semantic-Aware Discrete Absorbing Diffusion reconstructs the discrete token sequence ($x$) from a masked state ($\tilde{x}_t$). The Semantic Injection Interface (center) connects these topologies by projecting the sampled semantic plan $r$ into the decoder’s embedding space, conditioning the reverse diffusion step $p_\theta(x_{t-1} | \tilde{x}_t, r)$ to ensure global semantic coherence. Cross-Modal Alignment is applied to (b).
• Figure 3: Impact of the Semantic Injection Interface on convergence efficiency.CoM-DAD achieves superior BLEU scores with substantially reduced training costs compared to the ablated variant without the interface. The reported cost includes training of both the Continuous Latent Planner (Stage I) and Discrete Absorbing Diffusion (Stage II). These results indicate that the Semantic Injection Interface accelerates convergence by enabling the discrete model to exploit the structure of the continuous semantic manifold, rather than learning it from scratch.
• Figure 4: Analysis of parallel decoding dynamics and schedule impact. (a) Comparison of generation paradigms: CoM-DAD utilizes Discrete Absorbing Diffusion for efficient non-autoregressive parallel decoding, contrasting with the serial nature of autoregressive baselines. (b) Ablation on absorbing rates: Models trained with the aggressive Variable-Rate Noise Schedule (High Masking) demonstrate emergent semantic prioritization (main-first, details-later), establishing global structure before local details. Zoomed-in views are provided for clarity.
• Figure 5: Visualization of D-MLLM’s image generation capabilities. (a) Unconditional image generation results demonstrating diversity and visual quality. (b) Text-to-image generation showcasing effective cross-modal alignment, with synthesized images accurately reflecting the semantic content of input text prompts.