Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting

Chengyou Jia, Changliang Xia, Zhuohang Dang, Weijia Wu, Hangwei Qian, Minnan Luo

TL;DR

This paper proposes Automatic T2I generation, which aims to automate these tedious steps, allowing users to simply describe their needs in a freestyle chatting way, and proposes ChatGen-Evo, a multi-stage evolution strategy that progressively equips models with essential automation skills.

Abstract

Despite the significant advancements in text-to-image (T2I) generative models, users often face a trial-and-error challenge in practical scenarios. This challenge arises from the complexity and uncertainty of tedious steps such as crafting suitable prompts, selecting appropriate models, and configuring specific arguments, making users resort to labor-intensive attempts for desired images. This paper proposes Automatic T2I generation, which aims to automate these tedious steps, allowing users to simply describe their needs in a freestyle chatting way. To systematically study this problem, we first introduce ChatGenBench, a novel benchmark designed for Automatic T2I. It features high-quality paired data with diverse freestyle inputs, enabling comprehensive evaluation of automatic T2I models across all steps. Additionally, recognizing Automatic T2I as a complex multi-step reasoning task, we propose ChatGen-Evo, a multi-stage evolution strategy that progressively equips models with essential automation skills. Through extensive evaluation across step-wise accuracy and image quality, ChatGen-Evo significantly enhances performance over various baselines. Our evaluation also uncovers valuable insights for advancing automatic T2I. All our data, code, and models will be available in \url{https://chengyou-jia.github.io/ChatGen-Home}

ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting

TL;DR

This paper proposes Automatic T2I generation, which aims to automate these tedious steps, allowing users to simply describe their needs in a freestyle chatting way, and proposes ChatGen-Evo, a multi-stage evolution strategy that progressively equips models with essential automation skills.

Abstract

Despite the significant advancements in text-to-image (T2I) generative models, users often face a trial-and-error challenge in practical scenarios. This challenge arises from the complexity and uncertainty of tedious steps such as crafting suitable prompts, selecting appropriate models, and configuring specific arguments, making users resort to labor-intensive attempts for desired images. This paper proposes Automatic T2I generation, which aims to automate these tedious steps, allowing users to simply describe their needs in a freestyle chatting way. To systematically study this problem, we first introduce ChatGenBench, a novel benchmark designed for Automatic T2I. It features high-quality paired data with diverse freestyle inputs, enabling comprehensive evaluation of automatic T2I models across all steps. Additionally, recognizing Automatic T2I as a complex multi-step reasoning task, we propose ChatGen-Evo, a multi-stage evolution strategy that progressively equips models with essential automation skills. Through extensive evaluation across step-wise accuracy and image quality, ChatGen-Evo significantly enhances performance over various baselines. Our evaluation also uncovers valuable insights for advancing automatic T2I. All our data, code, and models will be available in \url{https://chengyou-jia.github.io/ChatGen-Home}

Paper Structure

This paper contains 34 sections, 6 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Text-to-Image Generation
  4. LLMs for Text-to-Image Generation
  5. Methodology
  6. ChatGenBench: Benchmarking Automatic T2I
  7. High-Quality Human Demonstration Collection
  8. LLM-Driven Role-Play for Chatting Generation
  9. Benchmark Construction
  10. Benchmark Analysis
  11. ChatGen: Achieving Automatic T2I
  12. ChatGen-Base with SFT
  13. ChatGen-Evo
  14. Stage 1: Prompt Writing via SFT.
  15. Stage 2: Model Selection via ModelToken.
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Illustration of tedious steps in T2I. Our method can select an appropriate model with suitable prompts and arguments. Note: Arle is a character from the game Genshin Impact.
  • Figure 2: Illustration of the data collection pipeline.
  • Figure 3: Illustration of the framework for ChatGen-Base and ChatGen-Evo.
  • Figure 4: User study results of ChatGen-Base and ChatGen-Evo.
  • Figure 5: Examples of images generated by different methods. Three rows represent single, multi-modal and historical inputs, respectively.
  • ...and 3 more figures