Generative Distribution Prediction: A Unified Approach to Multimodal Learning

TL;DR

GDP addresses the challenge of integrating heterogeneous multimodal data by learning the data-generating distribution with conditional synthetic data from diffusion-based generators. It combines transfer learning with dual-level embeddings and a unified diffusion backbone to produce point predictions and predictive distributions across tabular and unstructured modalities, while providing theoretical guarantees on excess risk via Wasserstein-distance bounds. The framework is validated on four tasks—domain adaptation, image captioning, Q&A, and adaptive quantile regression—demonstrating competitive or superior predictive accuracy and robust domain transfer. This work offers a scalable, distribution-focused approach for multimodal analytics, with practical impact in settings where heterogeneous data and domain shifts are prevalent.

Abstract

Accurate prediction with multimodal data-encompassing tabular, textual, and visual inputs or outputs-is fundamental to advancing analytics in diverse application domains. Traditional approaches often struggle to integrate heterogeneous data types while maintaining high predictive accuracy. We introduce Generative Distribution Prediction (GDP), a novel framework that leverages multimodal synthetic data generation-such as conditional diffusion models-to enhance predictive performance across structured and unstructured modalities. GDP is model-agnostic, compatible with any high-fidelity generative model, and supports transfer learning for domain adaptation. We establish a rigorous theoretical foundation for GDP, providing statistical guarantees on its predictive accuracy when using diffusion models as the generative backbone. By estimating the data-generating distribution and adapting to various loss functions for risk minimization, GDP enables accurate point predictions across multimodal settings. We empirically validate GDP on four supervised learning tasks-tabular data prediction, question answering, image captioning, and adaptive quantile regression-demonstrating its versatility and effectiveness across diverse domains.

Figures (4)

• Figure 1: Conditional diffusion models for target domain adaptation through shared representations.
• Figure 2: Comparison of star rating distributions between non-enthusiastic reviewers (source, blue) and enthusiastic (target, yellow) reviewers.
• Figure 3: Cosine similarity scores for the diffusion method vs. the BLIP model on a test set of 5,054 images. The left panel shows the average similarity between BLIP- or diffusion-generated captions and reference captions across $m=10$ generated captions per image. The right panel shows the average similarity for BLIP-GDP- or diffusion-GDP-selected captions, chosen from the same set of $m=10$ diffusion-generated captions.
• Figure 4: Comparative boxplots of cosine similarity scores for three question-answering methods in Q&A tasks over the test sample of size 100. "Temperature=0.0," "Temperature=0.7," and "GDP" represent the deterministic, generative, and GDP approaches, respectively, based on answers generated by the LLaMA-3.1-8B-Instruct model for a given question.

Theorems & Definitions (8)

• Theorem 1: GDP's excessive risk
• Corollary 1: Quantile adaptation
• proof : Proof of Theorem \ref{['thm_gdi']}
• proof : Proof of Corollary \ref{['cor1']}:
• Lemma 1: Theorem 3 in shen1994convergence, Lemma 11 in tian2024enhancing
• Theorem 2: Conditional diffusion via transfer learning
• proof
• Corollary 2: Non-transfer conditional diffusion generation