Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning

TL;DR

This work tackles the data-quality problem in reinforcement learning for Multimodal LLM perception, where entropy and response diversity rapidly collapse during GRPO training. It introduces Syn-GRPO, a self-evolving framework comprising a data server that performs foreground-consistent image outpainting and a GRPO-driven workflow that supervises diversity via a diversity reward and smoothing, all operating asynchronously to maintain training efficiency. The method yields substantial improvements across REC, OVD, and ISR, with particularly strong gains on out-of-domain data, and demonstrates robust data-scaling behavior and increasingly complex synthesized samples over time. Together, these results indicate that online, self-evolving data synthesis can significantly enhance long-horizon RL for MLLM perception and scalability.

Abstract

RL (reinforcement learning) methods (e.g., GRPO) for MLLM (Multimodal LLM) perception ability has attracted wide research interest owing to its remarkable generalization ability. Nevertheless, existing reinforcement learning methods still face the problem of low data quality, where data samples cannot elicit diverse responses from MLLMs, thus restricting the exploration scope for MLLM reinforcement learning. Some methods attempt to mitigate this problem by imposing constraints on entropy, but none address it at its root. Therefore, to tackle this problem, this work proposes Syn-GRPO (Synthesis-GRPO), which employs an online data generator to synthesize high-quality training data with diverse responses in GRPO training. Specifically, Syn-GRPO consists of two components: (1) data server; (2) GRPO workflow. The data server synthesizes new samples from existing ones using an image generation model, featuring a decoupled and asynchronous scheme to achieve high generation efficiency. The GRPO workflow provides the data server with the new image descriptions, and it leverages a diversity reward to supervise the MLLM to predict image descriptions for synthesizing samples with diverse responses. Experiment results across three visual perception tasks demonstrate that Syn-GRPO improves the data quality by a large margin, achieving significant superior performance to existing MLLM perception methods, and Syn-GRPO presents promising potential for scaling long-term self-evolving RL. Our code is available at https://github.com/hqhQAQ/Syn-GRPO.

Figures (10)

• Figure 1: In GRPO training, the MLLM generates non-diverse answers with low entropy, restricting the exploration space of RL.
• Figure 2: Entropy collapse and diversity collapse of GRPO for Qwen2.5-VL-3B on the visual perception task (REC). Note that 20% training progress corresponds to one training epoch.
• Figure 3: Framework of Syn-GRPO. Syn-GRPO consists of a data server and a GRPO workflow. The data server generates new high-quality images with diverse responses, based on the original images and image descriptions $d_{i^*}$. The GRPO workflow predicts new image descriptions for the data server, and it employs a diversity reward $\mathbf{R}_{\rm diversity}(o_i)$ to supervise the generation of these descriptions.
• Figure 4: Samples of three visual perception tasks (REC, OVD, and ISR) with the bounding box annotations.
• Figure 5: Synthesized images of three visual perception tasks.