Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

Lirui Zhao, Yue Yang, Kaipeng Zhang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, Rongrong Ji

TL;DR

DiffAgent tackles the challenge of selecting among thousands of text-to-image APIs by treating API choice as an LLM-driven tool-usage task. It introduces DABench to curate instruction-API pairs from the T2I ecosystem and a two-stage SFTA training regime (SFT followed by RRHF) to align API selection with human preferences, measured by a unified score combining CLIP, ImageReward, and HPS v2 signals. Empirical results show that DiffAgent-RRHF significantly improves the unified metric and reduces hallucinations compared to SFT-only baselines, while maintaining efficient inference times (~$4.81$s per instance). The work demonstrates that LLM agents can rapidly identify high-quality T2I APIs tailored to user prompts, with strong human-alignment signals and practical implications for tool-use in vision-language tasks.

Abstract

Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research. For example, the Civitai community, a platform for T2I innovation, currently hosts an impressive array of 74,492 distinct models. However, this diversity presents a formidable challenge in selecting the most appropriate model and parameters, a process that typically requires numerous trials. Drawing inspiration from the tool usage research of large language models (LLMs), we introduce DiffAgent, an LLM agent designed to screen the accurate selection in seconds via API calls. DiffAgent leverages a novel two-stage training framework, SFTA, enabling it to accurately align T2I API responses with user input in accordance with human preferences. To train and evaluate DiffAgent's capabilities, we present DABench, a comprehensive dataset encompassing an extensive range of T2I APIs from the community. Our evaluations reveal that DiffAgent not only excels in identifying the appropriate T2I API but also underscores the effectiveness of the SFTA training framework. Codes are available at https://github.com/OpenGVLab/DiffAgent.

DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

TL;DR

DiffAgent tackles the challenge of selecting among thousands of text-to-image APIs by treating API choice as an LLM-driven tool-usage task. It introduces DABench to curate instruction-API pairs from the T2I ecosystem and a two-stage SFTA training regime (SFT followed by RRHF) to align API selection with human preferences, measured by a unified score combining CLIP, ImageReward, and HPS v2 signals. Empirical results show that DiffAgent-RRHF significantly improves the unified metric and reduces hallucinations compared to SFT-only baselines, while maintaining efficient inference times (~s per instance). The work demonstrates that LLM agents can rapidly identify high-quality T2I APIs tailored to user prompts, with strong human-alignment signals and practical implications for tool-use in vision-language tasks.

Abstract

Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research. For example, the Civitai community, a platform for T2I innovation, currently hosts an impressive array of 74,492 distinct models. However, this diversity presents a formidable challenge in selecting the most appropriate model and parameters, a process that typically requires numerous trials. Drawing inspiration from the tool usage research of large language models (LLMs), we introduce DiffAgent, an LLM agent designed to screen the accurate selection in seconds via API calls. DiffAgent leverages a novel two-stage training framework, SFTA, enabling it to accurately align T2I API responses with user input in accordance with human preferences. To train and evaluate DiffAgent's capabilities, we present DABench, a comprehensive dataset encompassing an extensive range of T2I APIs from the community. Our evaluations reveal that DiffAgent not only excels in identifying the appropriate T2I API but also underscores the effectiveness of the SFTA training framework. Codes are available at https://github.com/OpenGVLab/DiffAgent.
Paper Structure (28 sections, 11 equations, 5 figures, 2 tables)

This paper contains 28 sections, 11 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. DABench
  5. APIs Collection
  6. APIs Filtering
  7. Description Reconstrcution
  8. Dataset composition
  9. DiffAgent
  10. DiffAgent-SFT
  11. DiffAgent-RRHF
  12. Human Preferences Evaluation
  13. Alignment with Human Preferences
  14. Experiment
  15. Experimental Settings
  16. ...and 13 more sections

Figures (5)

  • Figure 1: The comparison of different ways to T2I generation.
  • Figure 2: The data collection process of DABench. The left side is the data distributions of the data source, Civitai 2022civitai. These distributions include: (a) Model type. (b) Base model architecture. (c) Not suitable for work (NSFW) content. (d) File availability.
  • Figure 3: The training framework SFTA to get DiffAgent.
  • Figure 4: The visualization comparison between the original SD model (left) with DiffAgent's T2I API (right). The two lines come from the SD XL and SD 1.5 architectures respectively. For each pair, we provide the user prompt and the model name from T2I API.
  • Figure 5: The user study results of DiffAgent. Win rates are calculated without considering tie samples. DiffAgent surpasses the baseline in both relevance with prompt and human preference.