Teaching Text-to-Image Models to Communicate in Dialog

TL;DR

Dialog-to-image generation enables conversational agents to respond with images coherent to multi-turn dialogue. The authors propose a fine-tuning framework that preserves dialog structure through a '#' prefix concatenation and leverages in-domain data to reduce style mismatch, while freezing encoders and training a transformer-based diffusion noise predictor on top of pre-trained backbones like UniDiffuser and U-ViT. Across PhotoChat and MMDialog, the method yields consistent improvements in FID, IS, and CLIP-I over multiple backbones, with qualitative gains in facial realism and contextual alignment. This approach demonstrates that task-specific fine-tuning can adapt strong text-to-image models to dialog-driven image generation with modest compute, enabling more natural image-based responses in chat systems.

Abstract

A picture is worth a thousand words, thus, it is crucial for conversational agents to understand, perceive, and effectively respond with pictures. However, we find that directly employing conventional image generation techniques is inadequate for conversational agents to produce image responses effectively. In this paper, we focus on the innovative dialog-to-image generation task, where the model synthesizes a high-resolution image aligned with the given dialog context as a response. To tackle this problem, we design a tailored fine-tuning approach on the top of state-of-the-art text-to-image generation models to fully exploit the structural and semantic features in dialog context during image generation. Concretely, we linearize the dialog context with specific indicators to maintain the dialog structure, and employ in-domain data to alleviate the style mismatch between dialog-to-image and conventional image generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that our approach brings consistent and remarkable improvement with 3 state-of-the-art pre-trained text-to-image generation backbones.

Figures (5)

• Figure 1: An example of human conversations from PhotoChat Corpus zang-etal-2021-photochat. The speakers are talking about a grand reopening of a resort.
• Figure 2: An image is generated by the DALL$\cdot$E with the dialog context depicted in Figure 1 serving as its input.
• Figure 3: Implementation of the diffusion model with transformer backbone on dialog-image data.
• Figure 4: Different ways to indicate the dialogue structures.
• Figure 5: The IS and FID scores of 50 samples each belonging to the categories of people, food, and animals from PhotoChat test set.