Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding

TL;DR

EDDPMs generalize diffusion by introducing an adaptive encoder-decoder at the initial diffusion step, enabling simultaneous generation, reconstruction, and compact representation across text, images, and proteins. The framework retains the DDPM training objective while adding an alignment loss ${ m L}_{ ext{align}}$ and a reconstruction loss ${ m L}_{ ext{rec}}$, yielding a unified end-to-end model that learns a meaningful latent space. Empirical results across text, image, and protein domains show strong performance on generation, reconstruction, interpolation, and editing, with robust representation quality that supports downstream tasks like fitness prediction in proteins. The approach bridges VAEs, diffusion models, and latent-diffusion paradigms, offering a flexible, scalable foundation for multi-modal generative modeling with broad potential for future foundation-model development.

Abstract

The vast applications of deep generative models are anchored in three core capabilities -- generating new instances, reconstructing inputs, and learning compact representations -- across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce Generalized Encoding-Decoding Diffusion Probabilistic Models (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models.

Figures (14)

• Figure 1: (a) Generation, reconstruction, and representation are the core capabilities for diverse applications. (b) EDDPM shows comprehensive abilities on different text tasks in the customer-review domain (§\ref{['sec:exp_text']}).
• Figure 2: Different families of deep generative models.
• Figure 3: Training curves. EDDPM converges fast in a stable way. LatentOps, which depends on text VAE, relies on periodic scheduling of the weight of the prior regularization. The scheduling is common for training text VAE models DBLP:conf/emnlp/LiGLPLZG20, and leads to the ups-and-downs in the loss curve, indicating a difficult trade-off between generation and reconstruction capabilities.
• Figure 4: Overall performance comparison on images. SGXL stands for StyleGAN-XL. We stack the FIDs of generation, reconstruction, and interpolation together to show the overall performances of different models.
• Figure 5: Image Manipulation: The procedure for manipulation is detailed in Section \ref{['sec:exp_img:manipulation']}. The class names provided at the bottom indicate the target class towards which the images are being manipulated.