Is Your Text-to-Image Model Robust to Caption Noise?

TL;DR

This work analyzes how caption hallucination in Vision-Language Models used to recaption image data affects text-to-image generation. It builds the CCSBU benchmark with captions from multiple captioners to quantify the impact on generation quality and learned representations, and demonstrates that VLM-derived confidence signals correlate with hallucinated content. The authors propose a simple, effective mitigation by reweighting caption tokens during training based on VLM confidence, aided by a tokenizer-mapping step and attention-reweighting in CrossAttention. Empirical results show improvements in CLIP-Score, FID, and instruction-following benchmarks, and better intermediate representations, highlighting the importance of caption fidelity for robust T2I systems and suggesting avenues for further robust training strategies.

Abstract

In text-to-image (T2I) generation, a prevalent training technique involves utilizing Vision Language Models (VLMs) for image re-captioning. Even though VLMs are known to exhibit hallucination, generating descriptive content that deviates from the visual reality, the ramifications of such caption hallucinations on T2I generation performance remain under-explored. Through our empirical investigation, we first establish a comprehensive dataset comprising VLM-generated captions, and then systematically analyze how caption hallucination influences generation outcomes. Our findings reveal that (1) the disparities in caption quality persistently impact model outputs during fine-tuning. (2) VLMs confidence scores serve as reliable indicators for detecting and characterizing noise-related patterns in the data distribution. (3) even subtle variations in caption fidelity have significant effects on the quality of learned representations. These findings collectively emphasize the profound impact of caption quality on model performance and highlight the need for more sophisticated robust training algorithm in T2I. In response to these observations, we propose a approach leveraging VLM confidence score to mitigate caption noise, thereby enhancing the robustness of T2I models against hallucination in caption.

Figures (11)

• Figure 1: Hallucinated contents exist in image captions. We propose to use pre-trained vision-language models to provide signals in guiding text-to-image models to focus on more reliable terms.
• Figure 2: Histogram of token confidence score. The blue bars illustrate the confidence score distribution across all tokens, while the red bars focus specifically on hallucinated tokens. The dashed line denotes the mean confidence score of the corresponding tokens. The differential token confidence scores, obtained by subtracting the text-only score from the text-and-image score. The histogram of hallucinated tokens exhibits subtle but distinct variations compared to clean tokens, indicating measurable differences in their distributional characteristics.
• Figure 3: An overview of our proposed method to alleviate caption noise using a pre-trained VLM. We use a pre-trained VLM to compute the confidence score of each token in the caption. And then we use confidence score to reweight each token in T2I training, alleviating the effect of possibly hallucinated contents in captions.
• Figure 4: Illustration of tokenizer mapping. Given two distinct tokenizers, this tokenizer mapping allows us to construct a mapping function from one encoder output to another.
• Figure 5: We visualize the generated images and their text prompts across three models on the InstructBench. The top row represents outputs from the original Stable Diffusion 2.1-base model. The middle row is our base model, finetuned on the caption dataset without specific mitigation strategies for caption hallucination. The last row showcases our model trained with the proposed robust training framework. We split the generated images into four dimension: color, spatial, quantity, and feature. We observe that our method better follows the text prompts.