Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation

Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, Xueqian Wang

TL;DR

Chunk-GRPO introduces chunk-level reinforcement learning for flow-matching-based text-to-image generation to address inaccurate stepwise advantage attribution and overlooked temporal dynamics. By grouping timesteps into temporally guided chunks and optimizing at the chunk level, the method yields smoother gradients and better alignment with generation dynamics. An optional weighted sampling strategy further emphasizes high-noise chunks to boost preference alignment. Empirical results show consistent improvements over prior GRPO methods in preference metrics and image quality, proving the value of incorporating temporal structure into chunk-based optimization.

Abstract

Group Relative Policy Optimization (GRPO) has shown strong potential for flow-matching-based text-to-image (T2I) generation, but it faces two key limitations: inaccurate advantage attribution, and the neglect of temporal dynamics of generation. In this work, we argue that shifting the optimization paradigm from the step level to the chunk level can effectively alleviate these issues. Building on this idea, we propose Chunk-GRPO, the first chunk-level GRPO-based approach for T2I generation. The insight is to group consecutive steps into coherent 'chunk's that capture the intrinsic temporal dynamics of flow matching, and to optimize policies at the chunk level. In addition, we introduce an optional weighted sampling strategy to further enhance performance. Extensive experiments show that ChunkGRPO achieves superior results in both preference alignment and image quality, highlighting the promise of chunk-level optimization for GRPO-based methods.

Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation

TL;DR

Chunk-GRPO introduces chunk-level reinforcement learning for flow-matching-based text-to-image generation to address inaccurate stepwise advantage attribution and overlooked temporal dynamics. By grouping timesteps into temporally guided chunks and optimizing at the chunk level, the method yields smoother gradients and better alignment with generation dynamics. An optional weighted sampling strategy further emphasizes high-noise chunks to boost preference alignment. Empirical results show consistent improvements over prior GRPO methods in preference metrics and image quality, proving the value of incorporating temporal structure into chunk-based optimization.

Abstract

Group Relative Policy Optimization (GRPO) has shown strong potential for flow-matching-based text-to-image (T2I) generation, but it faces two key limitations: inaccurate advantage attribution, and the neglect of temporal dynamics of generation. In this work, we argue that shifting the optimization paradigm from the step level to the chunk level can effectively alleviate these issues. Building on this idea, we propose Chunk-GRPO, the first chunk-level GRPO-based approach for T2I generation. The insight is to group consecutive steps into coherent 'chunk's that capture the intrinsic temporal dynamics of flow matching, and to optimize policies at the chunk level. In addition, we introduce an optional weighted sampling strategy to further enhance performance. Extensive experiments show that ChunkGRPO achieves superior results in both preference alignment and image quality, highlighting the promise of chunk-level optimization for GRPO-based methods.
Paper Structure (21 sections, 1 theorem, 40 equations, 10 figures, 6 tables)

This paper contains 21 sections, 1 theorem, 40 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Action Chunk
  4. Reinforcement Learning for Diffusion-based Image Generation
  5. Preliminary
  6. Flow Matching
  7. GRPO on Flow Matching
  8. Method
  9. Chunk-level Optimization for GRPO
  10. chunk with temporal dynamics
  11. Chunk-GRPO
  12. Experiments
  13. Experiment Setup
  14. Main Results
  15. Ablation Study
  16. ...and 6 more sections

Key Result

Proposition 1

When advantage attribution is inaccurate at individual timesteps, optimizing them jointly within chunk yields better performance than optimizing them independently as steps, especially when the chunk size is small(e.g. a chunk size of 5).

Figures (10)

  • Figure 1: Chunk-GRPO significantly improves image quality, particularly in structure, lighting, and fine-grained details, demonstrating the superiority of chunk-level optimization.
  • Figure 2: While Trajectory1 has the greater final reward (advantage), its $t=1$ timestep is worse than that in Trajectory2. However, GRPO assigns the final advantages equally across all timesteps.
  • Figure 3: The prompt-invariant temporal dynamics of flow matching.
  • Figure 4: The overall framework of Chunk-GRPO. Chunk-GRPO integrates chunk-level optimization with temporal-dynamic-guided chunking, based on the grounded defined chunk-level importance ratio $r$. It also introduces an optional weighted sampling strategy, assigning sampling weight $w$ for each chunk.
  • Figure 5: Performance varies with different chunk sizes. The 'TD' refers to temporal dynamics.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Proposition 1