Prompt-Consistency Image Generation (PCIG): A Unified Framework Integrating LLMs, Knowledge Graphs, and Controllable Diffusion Models

TL;DR

Prompt-Consistency Image Generation (PCIG) presents a diffusion-based pipeline that dramatically improves alignment between textual prompts and generated images. By leveraging an LLM to extract objects and construct a knowledge graph, PCIG predicts accurate object locations and guides a controllable diffusion model, augmented with a visual text module and image-retrieval for proper-noun objects. The method explicitly targets four consistency facets—attribute, object, scene-text, and factual hallucinations—through integrated object extraction, relation extraction, and localization. Experimental results on MHaluBench show state-of-the-art performance across object-, text-, and fact-based hallucination metrics, underscoring the practical impact of combining LLMs, knowledge graphs, and controllable diffusion for reliable, prompt-faithful image synthesis.

Abstract

The rapid advancement of Text-to-Image(T2I) generative models has enabled the synthesis of high-quality images guided by textual descriptions. Despite this significant progress, these models are often susceptible in generating contents that contradict the input text, which poses a challenge to their reliability and practical deployment. To address this problem, we introduce a novel diffusion-based framework to significantly enhance the alignment of generated images with their corresponding descriptions, addressing the inconsistency between visual output and textual input. Our framework is built upon a comprehensive analysis of inconsistency phenomena, categorizing them based on their manifestation in the image. Leveraging a state-of-the-art large language module, we first extract objects and construct a knowledge graph to predict the locations of these objects in potentially generated images. We then integrate a state-of-the-art controllable image generation model with a visual text generation module to generate an image that is consistent with the original prompt, guided by the predicted object locations. Through extensive experiments on an advanced multimodal hallucination benchmark, we demonstrate the efficacy of our approach in accurately generating the images without the inconsistency with the original prompt. The code can be accessed via https://github.com/TruthAI-Lab/PCIG.

Figures (12)

• Figure 1: Selected samples generated by DALL-E 3. Each image represents one specific hallucination type. The inconsistency part for each image is highlighted in red.
• Figure 2: The pipeline of our PCIG method, using the example "A blue basketball jersey with the Golden State Warriors logo and 'Stephen Curry' written on it."
• Figure 3: Compared with multiple text-to-image generation methods. Our method shows comparable performance in all aspects.
• Figure 4: Ablation study on knowledge graph construction. Results become inaccurate in object locations when the proposed module is disable.
• Figure 5: Ablation study on object extraction. Results become inaccurate in object count and attribute when the proposed module is disable.