Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think

Zening Sun, Zhengpeng Xie, Lichen Bai, Shitong Shao, Shuo Yang, Zeke Xie

Abstract

Aligning Diffusion models has achieved remarkable breakthroughs in generating high-quality, human preference-aligned images. Existing techniques, such as supervised fine-tuning (SFT) and DPO-style preference optimization, have become principled tools for fine-tuning diffusion models. However, SFT relies on high-quality images that are costly to obtain, while DPO-style methods depend on large-scale preference datasets, which are often inconsistent in quality. Beyond data dependency, these methods are further constrained by computational inefficiency. To address these two challenges, we propose Composite Reward Assisted Fine-Tuning (CRAFT), a lightweight yet powerful fine-tuning paradigm that requires significantly reduced training data while maintaining computational efficiency. It first leverages a Composite Reward Filtering (CRF) technique to construct a high-quality and consistent training dataset and then perform an enhanced variant of SFT. We also theoretically prove that CRAFT actually optimizes the lower bound of group-based reinforcement learning, establishing a principled connection between SFT with selected data and reinforcement learning. Our extensive empirical results demonstrate that CRAFT with only 100 samples can easily outperform recent SOTA preference optimization methods with thousands of preference-paired samples. Moreover, CRAFT can even achieve 11-220$\times$ faster convergences than the baseline preference optimization methods, highlighting its extremely high efficiency.

CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think

Abstract

Aligning Diffusion models has achieved remarkable breakthroughs in generating high-quality, human preference-aligned images. Existing techniques, such as supervised fine-tuning (SFT) and DPO-style preference optimization, have become principled tools for fine-tuning diffusion models. However, SFT relies on high-quality images that are costly to obtain, while DPO-style methods depend on large-scale preference datasets, which are often inconsistent in quality. Beyond data dependency, these methods are further constrained by computational inefficiency. To address these two challenges, we propose Composite Reward Assisted Fine-Tuning (CRAFT), a lightweight yet powerful fine-tuning paradigm that requires significantly reduced training data while maintaining computational efficiency. It first leverages a Composite Reward Filtering (CRF) technique to construct a high-quality and consistent training dataset and then perform an enhanced variant of SFT. We also theoretically prove that CRAFT actually optimizes the lower bound of group-based reinforcement learning, establishing a principled connection between SFT with selected data and reinforcement learning. Our extensive empirical results demonstrate that CRAFT with only 100 samples can easily outperform recent SOTA preference optimization methods with thousands of preference-paired samples. Moreover, CRAFT can even achieve 11-220 faster convergences than the baseline preference optimization methods, highlighting its extremely high efficiency.
Paper Structure (37 sections, 1 theorem, 23 equations, 8 figures, 12 tables, 1 algorithm)

This paper contains 37 sections, 1 theorem, 23 equations, 8 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Diffusion Models.
  4. Aligning Diffusion Models.
  5. Methodology
  6. Formulation and Theoretical Insights
  7. Composite Reward Filtering
  8. Data construction.
  9. Data filtering.
  10. Composite Reward Assisted Fine-Tuning
  11. Experiment
  12. Experiment Setting
  13. Evaluation.
  14. Implementation Details.
  15. Main Results
  16. ...and 22 more sections

Key Result

Theorem 3.1

Given a Text-to-Image (T2I) diffusion model $p_\theta(\boldsymbol{x}_0|\boldsymbol{c})$, for any gradient direction $g$ with a small learning rateSuch assumption is reasonable, since most fine-tuning algorithms set the learning rate between 1e-5 and 1e-6.$\eta\rightarrow0$ ($\theta=\theta_{\mathrm{o where $\boldsymbol{\epsilon}_{\theta}$ is the noise predictive model of $p_{\theta}$, and $C$ is a

Figures (8)

  • Figure 1: Training Efficiency Comparison. Compared to SPO and SmPO, CRAFT reaches the same HPSv2.1 performance with 19.7× and 60.1× faster training time respectively, and further achieves superior final performance, demonstrating a dual advantage in both training speed and final generation quality.
  • Figure 2: Qualitative Improvements over Vanilla SDXL. We present CRAFT, an efficient and effective fine-tuning method designed to enhance diffusion models' alignment with human preference. The top row displays images generated by the base Vanilla-SDXL model, while the bottom row shows results generated by our CRAFT-finetuned model (CRAFT-SDXL). The comparison clearly shows that CRAFT-SDXL excels at complex instruction following and compositional reasoning, demonstrating significant superiority in adhering to diverse stylistic concepts (e.g., "Cyberpunk Style”), accurately generating specified objects and their attributes (e.g., the position of the cat "beside a rocket" and the teddy bear "wearing a helmet and cape"), and precisely rendering on-image text (e.g., "Deep Learning”).
  • Figure 3: The overall pipeline of our method mainly consists of three stages: (i) data construction, (ii) composite reward filtering, and (iii) weighted SFT fine-tuning.
  • Figure 4: Qualitative comparison. We provide a qualitative comparison of CRAFT-Diffusion and other different preference optimization methods (Vanilla, Diff-DPO, SPO) for SDXL. CRAFT-SDXL generates images of superior quality and prompt fidelity, showcasing significant improvements in detail, composition, and text rendering.
  • Figure 5: Winning Rate Comparison on Parti-Prompt. CRAFT outperforms all baseline methods in average winning rate against the base SDXL model.
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 3.1