Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling

TL;DR

Dynamic-TreeRPO tackles the inefficiencies of GRPO-based flow-matching in text-to-image generation by introducing a tree-structured, sliding-window sampling scheme with depth-wise dynamic noise, enabling diversified exploration while sharing computation across prefixes. It integrates LayerTuning-RL to fuse supervised fine-tuning with reinforcement learning, and introduces dynamic clipping bounds guided by trajectory rewards, mitigating exploration disruption. The approach yields faster convergence and higher-quality generations, outperforming state-of-the-art baselines on benchmarks like HPS-v2.1, PickScore, and ImageReward, and improves training efficiency by about 50%. This work advances efficient, robust exploration in flow-based diffusion models and offers a practical, scalable framework for RL-assisted T2I generation.

Abstract

The integration of Reinforcement Learning (RL) into flow matching models for text-to-image (T2I) generation has driven substantial advances in generation quality. However, these gains often come at the cost of exhaustive exploration and inefficient sampling strategies due to slight variation in the sampling group. Building on this insight, we propose Dynamic-TreeRPO, which implements the sliding-window sampling strategy as a tree-structured search with dynamic noise intensities along depth. We perform GRPO-guided optimization and constrained Stochastic Differential Equation (SDE) sampling within this tree structure. By sharing prefix paths of the tree, our design effectively amortizes the computational overhead of trajectory search. With well-designed noise intensities for each tree layer, Dynamic-TreeRPO can enhance the variation of exploration without any extra computational cost. Furthermore, we seamlessly integrate Supervised Fine-Tuning (SFT) and RL paradigm within Dynamic-TreeRPO to construct our proposed LayerTuning-RL, reformulating the loss function of SFT as a dynamically weighted Progress Reward Model (PRM) rather than a separate pretraining method. By associating this weighted PRM with dynamic-adaptive clipping bounds, the disruption of exploration process in Dynamic-TreeRPO is avoided. Benefiting from the tree-structured sampling and the LayerTuning-RL paradigm, our model dynamically explores a diverse search space along effective directions. Compared to existing baselines, our approach demonstrates significant superiority in terms of semantic consistency, visual fidelity, and human preference alignment on established benchmarks, including HPS-v2.1, PickScore, and ImageReward. In particular, our model outperforms SoTA by $4.9\%$, $5.91\%$, and $8.66\%$ on those benchmarks, respectively, while improving the training efficiency by nearly $50\%$.

Figures (8)

• Figure 1: Compare with the previous method. $Left$: The reward curve during training shows that Dynamic-TreeRPO converges more rapidly than both DanceGRPO and MixGRPO, and ultimately achieves significantly better results than either of them. $Right$: Visualization of the different structures. Dynamic-TreeRPO employs a tree structure with a sliding window mechanism. MixGRPO utilizes a sliding window structure, where SDE is applied only during the sliding window period. In contrast, DanceGRPO applies SDE throughout the entire process.
• Figure 2: The framework of Dynamic-TreeRPO. $(a)$ Dynamic Tree Structure. Noise intensity is dynamically introduced for the nodes of each layer in the tree structure. $(b)$ Paths to multiple group trees. For each path, the highest reward score is selected. $(c)$ PRM supervision of each node. The node with the maximum reward is used to supervise the model’s predictions at each layer. $(d)$ Training procedure of Dynamic-TreeRPO.
• Figure 3: Qualitative comparison. Dynamic-TreeRPO achieves superior performance compared to Flux,DanceGRPO and MixGRPO in terms of semantics, aesthetics and text-image alignment.
• Figure 4: Ablation Studies on reward sensitivity factor and balancing parameter in LayerTuning-RL.
• Figure 5: Comparison of the visualization results of FLUX, DanceGRPO, MixGRPO and Dynamic-TreeRPO.