Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptRL: Prompt Matters in RL for Flow-Based Image Generation

Fu-Yun Wang, Han Zhang, Michael Gharbi, Hongsheng Li, Taesung Park

TL;DR

PromptRL addresses two major inefficiencies in flow-based RL for text-to-image generation: limited exploration diversity and prompt overfitting. It achieves this by jointly training a language model to act as a dynamic prompt refiner within the RL loop, expanding the linguistic space while a flow-matching model learns from both original and LM-refined prompts. The framework employs a prompt retention mechanism, group-wise advantage normalization, and per-sample reward tagging to enable robust, multi-objective optimization without manual reward tuning. Empirically, PromptRL attains state-of-the-art GenEval (0.97), OCR accuracy (0.98), and PickScore (24.04), and notably improves image-editing EditReward to 1.43 with only 0.06M rollouts, while requiring up to 2× fewer samples than flow-only RL. The results indicate that language-vision co-optimization can markedly enhance exploration, learning efficiency, and generalization in reward-guided generative modeling.

Abstract

Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present PromptRL (Prompt Matters in RL for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2$\times$ fewer rollouts compared to naive flow-only RL. Our code is available at https://github.com/G-U-N/UniRL.

PromptRL: Prompt Matters in RL for Flow-Based Image Generation

TL;DR

PromptRL addresses two major inefficiencies in flow-based RL for text-to-image generation: limited exploration diversity and prompt overfitting. It achieves this by jointly training a language model to act as a dynamic prompt refiner within the RL loop, expanding the linguistic space while a flow-matching model learns from both original and LM-refined prompts. The framework employs a prompt retention mechanism, group-wise advantage normalization, and per-sample reward tagging to enable robust, multi-objective optimization without manual reward tuning. Empirically, PromptRL attains state-of-the-art GenEval (0.97), OCR accuracy (0.98), and PickScore (24.04), and notably improves image-editing EditReward to 1.43 with only 0.06M rollouts, while requiring up to 2× fewer samples than flow-only RL. The results indicate that language-vision co-optimization can markedly enhance exploration, learning efficiency, and generalization in reward-guided generative modeling.

Abstract

Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present PromptRL (Prompt Matters in RL for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2 fewer rollouts compared to naive flow-only RL. Our code is available at https://github.com/G-U-N/UniRL.
Paper Structure (17 sections, 4 equations, 5 figures, 9 tables, 1 algorithm)

This paper contains 17 sections, 4 equations, 5 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Understanding flow RL inefficiencies
  4. Generation dilemma: quality vs. diversity
  5. Prompt linguistic hacking
  6. PromptRL
  7. Incorporating LMs as dynamic prompt refiner
  8. Joint RL training on disjoint LMs and FMs
  9. Multi-reward training via reward tagging
  10. Experiments
  11. Experimental setup
  12. Quantitative comparison
  13. Qualitative comparison
  14. Ablation study
  15. Conclusion
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Overview of the PromptRL framework. PromptRL jointly trains a language model and a flow-matching image generator within a unified RL loop. Given an original prompt (and optionally a reference image), the LM produces semantically grounded prompt variants that expand the exploration space beyond fixed-prompt training. These prompts are paired with independent noise samples and passed to the flow-matching model to generate diverse images. A mixture of reward functions evaluates each image and guides the evolution of the LM (for improved prompt rewriting) and the FM (for improved visual generation).
  • Figure 2: The quality-diversity dilemma in flow-based T2I models and its mitigation through prompt refinement. As models advance from Stable Diffusion v1-5 (a) to FLUX.1-dev (b), they achieve higher text-image alignment (TI-Sim) and aesthetic quality (P.S.) but suffer from dramatically reduced output diversity (II-Sim), creating an exploration bottleneck for RL optimization. LM-based prompt refinement (c) partially restores diversity while maintaining quality, demonstrating that linguistic variations can expand the exploration space. All images in each row share identical random seeds to isolate the effect of prompt conditioning.
  • Figure 3: Qualitative comparison on instructional image editing tasks. Our method enables the LM to leverage the original image's visual signals to transform vague editing instructions into more explicit and image-specific prompts, ultimately improving editing performance.
  • Figure 4: Qualitative comparison on text-to-image generation. The first two prompts are from GenEval. The third prompt is from OCR-1k. And the last two prompts are from Drawbench.
  • Figure 5: Training curve comparison on different rewards.