DiffusionGPT: LLM-Driven Text-to-Image Generation System

TL;DR

This work tackles the challenge of diverse prompts and limited generalization in text-to-image generation by introducing DiffusionGPT, an LLM-driven, unified framework that orchestrates domain-specific diffusion variants. It builds a Tree-of-Thought over available generators, uses prompt parsing and a human-informed Advantage Database to select the most suitable component, and executes generation with optional prompt extension. The approach is training-free and plug-and-play, with extensive experiments showing improvements in semantic alignment and aesthetics over baseline diffusion models, confirmed by a user study. The results highlight a practical pathway to scalable, cross-domain image synthesis that can incorporate new domain experts with minimal overhead.

Abstract

Diffusion models have opened up new avenues for the field of image generation, resulting in the proliferation of high-quality models shared on open-source platforms. However, a major challenge persists in current text-to-image systems are often unable to handle diverse inputs, or are limited to single model results. Current unified attempts often fall into two orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate expert model to output. To combine the best of both worlds, we propose DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified generation system capable of seamlessly accommodating various types of prompts and integrating domain-expert models. DiffusionGPT constructs domain-specific Trees for various generative models based on prior knowledge. When provided with an input, the LLM parses the prompt and employs the Trees-of-Thought to guide the selection of an appropriate model, thereby relaxing input constraints and ensuring exceptional performance across diverse domains. Moreover, we introduce Advantage Databases, where the Tree-of-Thought is enriched with human feedback, aligning the model selection process with human preferences. Through extensive experiments and comparisons, we demonstrate the effectiveness of DiffusionGPT, showcasing its potential for pushing the boundaries of image synthesis in diverse domains.

Figures (9)

• Figure 1: We propose a unified generation system DiffusionGPT, which leverages Large Language Models (LLM) to seamlessly accommodating various types of prompts input and integrating domain-expert models for output. Our system is capable of parsing diverse forms of inputs, including Prompt-based, Instruction-based, Inspiration-based, and Hypothesis-based input types. It exhibits the ability to generate outputs of superior quality.
• Figure 2: Overview of DiffusionGPT. The workflow of DiffusionGPT consists of four steps: Prompt Parse, Tree-of-thought of Models of Building and Searching, Model Selection, and Execution Generation. The four steps are shown from left to right and interact with LLM continuously. The upper side shows the detailed process of each step. The lower side shows the example of the whole workflow.
• Figure 3: Details of prompts during interactions with the ChatGPTinstructGPT. Before being inputted into the ChatGPT, the slots "{}" in figure are uniformly replaced with the corresponding text values.
• Figure 4: When comparing SD1.5-based DiffusionGPT with SD15SD15, it is observed that DiffusionGPT excels in generating more realistic results at a fine-grained level for categories such as humans and scenes. The generated images demonstrate improved visual fidelity, capturing finer details and exhibiting a higher degree of realism compared to SD15.
• Figure 5: Comparison of SDXL version of DiffusionGPT with baseline SDXLsdxl. All generated iamges are 1024$\times$1024 pixels.