Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation

Ziwei Huang, Ying Shu, Hao Fang, Quanyu Long, Wenya Wang, Qiushi Guo, Tiezheng Ge, Leilei Gan

TL;DR

Subject-driven image generation faces a fundamental fidelity vs. editability trade-off. The paper introduces Customized-GRPO, a reinforcement-learning framework that combines Synergy-Aware Reward Shaping (SARS) and Time-Aware Dynamic Weighting (TDW) to address reward conflicts and align optimization with diffusion timesteps. Empirical results on DreamBench show state-of-the-art balance between identity preservation and prompt following, with notable gains in DINO, CLIP-I, and HPS-v3, validated by ablations and human evaluations. The approach offers a practical, model-agnostic pathway to improve multi-objective RL in diffusion-based generation and potentially generalizes to other architectures and multi-subject setups.

Abstract

Subject-driven image generation models face a fundamental trade-off between identity preservation (fidelity) and prompt adherence (editability). While online reinforcement learning (RL), specifically GPRO, offers a promising solution, we find that a naive application of GRPO leads to competitive degradation, as the simple linear aggregation of rewards with static weights causes conflicting gradient signals and a misalignment with the temporal dynamics of the diffusion process. To overcome these limitations, we propose Customized-GRPO, a novel framework featuring two key innovations: (i) Synergy-Aware Reward Shaping (SARS), a non-linear mechanism that explicitly penalizes conflicted reward signals and amplifies synergistic ones, providing a sharper and more decisive gradient. (ii) Time-Aware Dynamic Weighting (TDW), which aligns the optimization pressure with the model's temporal dynamics by prioritizing prompt-following in the early, identity preservation in the later. Extensive experiments demonstrate that our method significantly outperforms naive GRPO baselines, successfully mitigating competitive degradation. Our model achieves a superior balance, generating images that both preserve key identity features and accurately adhere to complex textual prompts.

From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation

TL;DR

Subject-driven image generation faces a fundamental fidelity vs. editability trade-off. The paper introduces Customized-GRPO, a reinforcement-learning framework that combines Synergy-Aware Reward Shaping (SARS) and Time-Aware Dynamic Weighting (TDW) to address reward conflicts and align optimization with diffusion timesteps. Empirical results on DreamBench show state-of-the-art balance between identity preservation and prompt following, with notable gains in DINO, CLIP-I, and HPS-v3, validated by ablations and human evaluations. The approach offers a practical, model-agnostic pathway to improve multi-objective RL in diffusion-based generation and potentially generalizes to other architectures and multi-subject setups.

Abstract

Subject-driven image generation models face a fundamental trade-off between identity preservation (fidelity) and prompt adherence (editability). While online reinforcement learning (RL), specifically GPRO, offers a promising solution, we find that a naive application of GRPO leads to competitive degradation, as the simple linear aggregation of rewards with static weights causes conflicting gradient signals and a misalignment with the temporal dynamics of the diffusion process. To overcome these limitations, we propose Customized-GRPO, a novel framework featuring two key innovations: (i) Synergy-Aware Reward Shaping (SARS), a non-linear mechanism that explicitly penalizes conflicted reward signals and amplifies synergistic ones, providing a sharper and more decisive gradient. (ii) Time-Aware Dynamic Weighting (TDW), which aligns the optimization pressure with the model's temporal dynamics by prioritizing prompt-following in the early, identity preservation in the later. Extensive experiments demonstrate that our method significantly outperforms naive GRPO baselines, successfully mitigating competitive degradation. Our model achieves a superior balance, generating images that both preserve key identity features and accurately adhere to complex textual prompts.
Paper Structure (52 sections, 13 equations, 11 figures, 5 tables)

This paper contains 52 sections, 13 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Subject-Driven Generation
  4. GRPO in Text-to-Image Generation
  5. GRPO in Autoregressive Models.
  6. GRPO in Diffusion and Flow Matching Models.
  7. GRPO in Diffusion and Flow Matching Models.
  8. Problem Formulation
  9. Diffusion Model.
  10. Flow Matching.
  11. Subject-Driven Image Generation.
  12. Optimization with Reinforcement Learning.
  13. Pilot Experiments and Analysis
  14. Method
  15. Synergy-Aware Reward Shaping
  16. ...and 37 more sections

Figures (11)

  • Figure 1: Customized-GRPO achieves a state-of-the-art balance between ID Preservation and Prompt Following on DreamBench. Our method and Naive GRPO are built upon UNO.
  • Figure 2: Comparison of Naive GRPO with fixed linear weights against our Customized-GRPO. The (a, b) training curves show that Naive GRPO with 1:1 weights (red) overfits to identity, causing a collapse in prompt following while 1:1.5 weights (purple) prevents this collapse at the cost of poor identity learning. Our method (green) improves both capabilities concurrently. The qualitative results (bottom) confirm these findings.
  • Figure 3: Cross-section of the SARS function, illustrating how it reshapes the linear reward landscape. We plot the final advantage as a function of $A_{\text{id}}$ while holding $A_{\text{prompt}}$ constant at 1.0.
  • Figure 4: Temporal Dynamics of Feature Synthesis during the Denoising Process. (Right) The plots show that low-frequency energy (structure) converges early, whereas high-frequency energy (details) accumulates steadily over all phases. (Left) The visual progression from pure noise (t=T) to the final image (t=0) corroborates this finding.
  • Figure 5: Human evaluation. Pairwise comparison between our Customized-GRPO and the base model.
  • ...and 6 more figures