Information Theoretic Text-to-Image Alignment
Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro Michiardi
TL;DR
This work tackles the challenge of aligning diffusion-based Text-to-Image models with user intent by introducing a self-supervised, information-theoretic signal. It defines a point-wise mutual information estimator $I(z,p)$ that quantifies how much a prompt $p$ informs the denoising process of an image latent $z$ via a pre-trained denoiser, without relying on auxiliary models. Leveraging this signal, MITUNE builds a synthetic, MI-filtered fine-tuning set from model-generated data and applies parameter-efficient adapter-based fine-tuning to the denoising network, achieving state-of-the-art alignment across standard benchmarks while maintaining image quality. The method is data-efficient, self-contained, and adaptable to different base diffusion models, with clear insights into the trade-offs between alignment and image diversity through CFG adjustments.
Abstract
Diffusion models for Text-to-Image (T2I) conditional generation have recently achieved tremendous success. Yet, aligning these models with user's intentions still involves a laborious trial-and-error process, and this challenging alignment problem has attracted considerable attention from the research community. In this work, instead of relying on fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language models, we use Mutual Information (MI) to guide model alignment. In brief, our method uses self-supervised fine-tuning and relies on a point-wise (MI) estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment. Our analysis indicates that our method is superior to the state-of-the-art, yet it only requires the pre-trained denoising network of the T2I model itself to estimate MI, and a simple fine-tuning strategy that improves alignment while maintaining image quality. Code available at https://github.com/Chao0511/mitune.