Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation

Jingkun An, Yinghao Zhu, Zongjian Li, Enshen Zhou, Haoran Feng, Xijie Huang, Bohua Chen, Yemin Shi, Chengwei Pan

TL;DR

AGFSync introduces a fully AI-driven framework that uses Vision-Language Models to generate multi-aspect feedback and direct preference optimization (DPO) to align text-to-image diffusion models without human annotations. By constructing AI-generated prompts, QA pairs, and a weighted quality score combining VQA, CLIP, and aesthetics, AGFSync finetunes SD v1.4, v1.5, and SDXL-base to improve prompt fidelity and image aesthetics. Across TIFA and HPS v2 benchmarks, AGFSync achieves consistent gains in VQA fidelity, CLIP alignment, and aesthetic judgments, and shows strong alignment with human preferences via GPT-4V and human evaluators. The approach demonstrates scalable, data-efficient diffusion-model alignment with publicly available code and datasets, signaling practical impact for robust and scalable T2I systems.

Abstract

Text-to-Image (T2I) diffusion models have achieved remarkable success in image generation. Despite their progress, challenges remain in both prompt-following ability, image quality and lack of high-quality datasets, which are essential for refining these models. As acquiring labeled data is costly, we introduce AGFSync, a framework that enhances T2I diffusion models through Direct Preference Optimization (DPO) in a fully AI-driven approach. AGFSync utilizes Vision-Language Models (VLM) to assess image quality across style, coherence, and aesthetics, generating feedback data within an AI-driven loop. By applying AGFSync to leading T2I models such as SD v1.4, v1.5, and SDXL-base, our extensive experiments on the TIFA dataset demonstrate notable improvements in VQA scores, aesthetic evaluations, and performance on the HPSv2 benchmark, consistently outperforming the base models. AGFSync's method of refining T2I diffusion models paves the way for scalable alignment techniques. Our code and dataset are publicly available at https://anjingkun.github.io/AGFSync.

AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation

TL;DR

AGFSync introduces a fully AI-driven framework that uses Vision-Language Models to generate multi-aspect feedback and direct preference optimization (DPO) to align text-to-image diffusion models without human annotations. By constructing AI-generated prompts, QA pairs, and a weighted quality score combining VQA, CLIP, and aesthetics, AGFSync finetunes SD v1.4, v1.5, and SDXL-base to improve prompt fidelity and image aesthetics. Across TIFA and HPS v2 benchmarks, AGFSync achieves consistent gains in VQA fidelity, CLIP alignment, and aesthetic judgments, and shows strong alignment with human preferences via GPT-4V and human evaluators. The approach demonstrates scalable, data-efficient diffusion-model alignment with publicly available code and datasets, signaling practical impact for robust and scalable T2I systems.

Abstract

Text-to-Image (T2I) diffusion models have achieved remarkable success in image generation. Despite their progress, challenges remain in both prompt-following ability, image quality and lack of high-quality datasets, which are essential for refining these models. As acquiring labeled data is costly, we introduce AGFSync, a framework that enhances T2I diffusion models through Direct Preference Optimization (DPO) in a fully AI-driven approach. AGFSync utilizes Vision-Language Models (VLM) to assess image quality across style, coherence, and aesthetics, generating feedback data within an AI-driven loop. By applying AGFSync to leading T2I models such as SD v1.4, v1.5, and SDXL-base, our extensive experiments on the TIFA dataset demonstrate notable improvements in VQA scores, aesthetic evaluations, and performance on the HPSv2 benchmark, consistently outperforming the base models. AGFSync's method of refining T2I diffusion models paves the way for scalable alignment techniques. Our code and dataset are publicly available at https://anjingkun.github.io/AGFSync.
Paper Structure (55 sections, 9 equations, 7 figures, 15 tables)

This paper contains 55 sections, 9 equations, 7 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aligning Diffusion Models Methods
  4. Image Quality Evaluation Methods
  5. Methodology
  6. Preference Candidate Set Generation
  7. Preference Pair Construction
  8. VQA Questions Generation
  9. VQA Score
  10. CLIP Score
  11. Aesthetic Score
  12. Weighted Score Calculation
  13. Preference Pair Dataset Construction
  14. DPO Alignment
  15. Experimental Setups
  16. ...and 40 more sections

Figures (7)

  • Figure 1: Overall pipeline of AGFSync, which mainly encompasses 3 steps. AGFSync learns from AI-generated feedback data with DPO. AGFSync requires no human annotation, model architecture changes, or reinforcement learning.
  • Figure 2: Comparison of the win rates of SD v1.4, SD v1.5 and SDXL-base with or without our AGFSync on HPS v2. CLIP score (left) and aesthetic score (right).
  • Figure 3: Impact of noise on image diversity. With the numbers on the left side of the images indicating the increasing weight of noise, four images were generated using the same text input "wild animal".
  • Figure 4: Comparison between SDXL-base and AGFSync (Ours)+SDXL-base. (a) Red-highlighted text indicates discrepancies with input prompts. (b) Third row compares details, showing AGFSync's improved coherence and detail.
  • Figure 5: We introduce AGFSync: a model-agnostic training algorithm that improves text-to-image (T2I) generation models' faithfulness and coherence to text inputs and image aesthetics without human interventions. The images showcase a comparison of the results before and after finetuning SDXL with AGFSync.
  • ...and 2 more figures