DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO

TL;DR

This work tackles mode collapse in GRPO-based image generation by redesigning both the reward and regularization to promote diversity without sacrificing quality. It introduces DiverseGRPO, combining a distributional creativity bonus—constructed via spectral clustering over samples from the same caption and informed by perceptual distances—and a structure-aware regularization that emphasizes early-stage diversity through a Wasserstein-based constraint. Across multiple backbones and reward models, the approach yields a 13–18% improvement in semantic diversity under matched quality, achieving a new Pareto frontier between diversity and fidelity. By expanding semantic coverage in perceptual space while preserving caption fidelity, DiverseGRPO offers a practical path to more creative diffusion-based image generation.

Abstract

Reinforcement learning (RL), particularly GRPO, improves image generation quality significantly by comparing the relative performance of images generated within the same group. However, in the later stages of training, the model tends to produce homogenized outputs, lacking creativity and visual diversity, which restricts its application scenarios. This issue can be analyzed from both reward modeling and generation dynamics perspectives. First, traditional GRPO relies on single-sample quality as the reward signal, driving the model to converge toward a few high-reward generation modes while neglecting distribution-level diversity. Second, conventional GRPO regularization neglects the dominant role of early-stage denoising in preserving diversity, causing a misaligned regularization budget that limits the achievable quality--diversity trade-off. Motivated by these insights, we revisit the diversity degradation problem from both reward modeling and generation dynamics. At the reward level, we propose a distributional creativity bonus based on semantic grouping. Specifically, we construct a distribution-level representation via spectral clustering over samples generated from the same caption, and adaptively allocate exploratory rewards according to group sizes to encourage the discovery of novel visual modes. At the generation level, we introduce a structure-aware regularization, which enforces stronger early-stage constraints to preserve diversity without compromising reward optimization efficiency. Experiments demonstrate that our method achieves a 13\%--18\% improvement in semantic diversity under matched quality scores, establishing a new Pareto frontier between image quality and diversity for GRPO-based image generation.

Figures (7)

• Figure 1: (a) Image generation models trained with GRPO suffer from mode collapse (similar faces, camera angles, etc.), which limits their applicability in creative scenarios. (b) The proposed DiverseGRPO method achieves higher diversity while maintaining comparable quality. (c) DiverseGRPO successfully maintains a healthier level of diversity across the entire duration of training, while the baseline method suffers from a premature collapse. (d) In the Inception feature space, DiverseGRPO generates images that cover a significantly broader range of semantic features, effectively mitigating mode collapse.
• Figure 2: Analysis of the reasons for mode collapse: (Left) Policy model collapse into high-reward modes due to single sample reward modeling. (Right) Conventional regularization neglects the dominant role of early-stage denoising in preserving diversity.
• Figure 3: DiverseGRPO employs two primary strategies to mitigate mode collapse: (a) A distributional creativity bonus mechanism based on semantic grouping. It begins by applying spectral clustering to images generated from the same caption, then assigns exploratory rewards according to cluster size to encourage the emergence of novel visual modes. (b) Structure-aware regularization imposes stronger constraints during the initial denoising stages to preserve sample diversity, while gradually relaxing the penalty in later stages to enhance the effectiveness of reward optimization.
• Figure 4: Qualitative experiments on diversity, the baseline method exhibits mode collapse in the generation of main subjects (such as facial features, poses, font colors, etc.), whereas our method achieves greater diversity and creativity while maintaining image quality and consistency with the captions.
• Figure 5: Ablation study on the Pareto front of quality and diversity for different modules and parameters.